Confusing Stats Collector Logs (`GetObject`)

[shirady]

Environment info

• NooBaa Version: current master (4.23)
• Platform: minikube

Actual behavior

1. After GetObject in the endpoint logs I see the counter is on 2 and the differences between the min and the max does not represent the same op.

Expected behavior

1. Logs without "nsfs" stats with the data not related to NSFS.
2. (not sure if it's by design) I think I would expect to see the same operation counted (getobject) and counted as 1.

Steps to reproduce

I found it during working on another feature, but it would probably be relevant when running get object on the default backingstore.

I worked in the following setup:

1. Deploy NooBaa on Minikube after adding GCP WIF configurations - see guide "Create GCP WIF (STS) Setup On Minikube With NooBaa" (link).
2. Create GCP WIF (STS) backing using the CLI nooba backingstore create google-cloud-storage-sts shira-gcp-sts -n test1
3. Create Bucketclass: nb bucketclass create placement-bucketclass bc-shira --backingstores=shira-gcp-sts -n test1
4. Create OBC: nb obc create shira-obc --bucketclass=bc-shira -n test1 --show-secrets
5. Use port-forward (different tab): kubectl port-forward service/s3 12443:443 -n test1
6. Alias S3 Client with the OBC details: alias s3-obc-user-1=‘AWS_SECRET_ACCESS_KEY=<secret-key> AWS_ACCESS_KEY_ID=<access-key> aws --no-verify-ssl --endpoint-url https://localhost:12443’
7. PutObject: echo 'test_data' | s3-obc-user-1 s3 cp - s3://shira-obc-48c04eb3-48d9-4042-9f59-24f16ea71d09/test_data1.txt
8. GetObject: touch my_file.txt s3-obc-user-1 s3api get-object --bucket shira-obc-48c04eb3-48d9-4042-9f59-24f16ea71d09 --key test_data.txt my_file.txt

More information - Screenshots / Logs / Other output

From the endpoint logs:
After put object - is it not in nsfs, but we can see "nsfs stats" in the logs.

Jun-30 5:41:22.823 [Endpoint/12]    [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT shira-obc-48c04eb3-48d9-4042-9f59-24f16ea71d09 test_data1.txt
Jun-30 5:41:36.329 [Endpoint/12]    [L0] core.sdk.endpoint_stats_collector:: nsfs stats - IO counters : undefined
Jun-30 5:41:36.329 [Endpoint/12]    [L0] core.sdk.endpoint_stats_collector:: nsfs stats - S3 op=upload_object : { count: 1, error_count: 0, min_time: 3503, max_time: 3503, sum_time: 3503 }
Jun-30 5:41:36.329 [Endpoint/12]    [L0] core.sdk.endpoint_stats_collector:: nsfs stats - S3 op=list_objects : { count: 1, error_count: 0, min_time: 5, max_time: 5, sum_time: 5 }

After get object - again, not in nsfs, but also counted as 2 in read_object (with a difference in times: min_time: 1 vs max_time: 38) - doesn't count the same operation.

Jun-30 5:46:18.090 [Endpoint/12]    [L0] core.sdk.endpoint_stats_collector:: nsfs stats - IO counters : undefined
Jun-30 5:46:18.091 [Endpoint/12]    [L0] core.sdk.endpoint_stats_collector:: nsfs stats - S3 op=read_object : { count: 2, error_count: 0, min_time: 1, max_time: 38, sum_time: 39 }
Jun-30 5:46:38.092 [Endpoint/12]    [L0] core.sdk.endpoint_stats_collector:: bucket stats - shira-obc-48c04eb3-48d9-4042-9f59-24f16ea71d09 text/plain : { read_count: 1 }
Jun-30 5:46:38.095 [Endpoint/12]   [LOG] CONSOLE:: update_endpoint_stats. namespace_stats = []
Jun-30 5:46:41.475 [Endpoint/12]    [L0] core.agent.block_store_services.block_store_client:: Sending io_stats {"read_count":1,"read_bytes":12,"write_count":0,"write_bytes":0} to block address n2n://6a434e2826c212001faca87e

[coderabbitai]

🔗 Related PRs

#9135 - Removed counters queries from md_store find_objects [merged]
#9171 - [NSFS | Glacier] Disallow UploadPartCopy if the source is not restored [merged]
#9529 - Fix: GCP namespaced bucket downloads failing with ECONNRESET [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!