Content left in input buffer

[yangliping]

This is a Lustre bug where a negative value appears in ost_read. I would like to request the following:

1. Add the unparsed content to the debug output.
2. Bypass negative values.

Could you please implement these changes in the lustrefs-exporter project? The negative value in Lustre's statistics is causing an issue in lustrefs-exporter when it collects information. Adding the unparsed content to debug output and ignoring negative values would help address this problem.

2025-05-21T09:16:32.708605Z  INFO lustrefs_exporter: Listening on http://0.0.0.0:9169/metrics
2025-05-21T09:16:47.586040Z DEBUG opentelemetry_sdk:  name="MeterProvider.Building" builder="MeterProviderBuilder { resource: Some(Resource { inner: ResourceInner { attrs: {Static(\"telemetry.sdk.name\"): String(Static(\"opentelemetry\")), Static(\"telemetry.sdk.language\"): String(Static(\"rust\")), Static(\"service.name\"): String(Static(\"lustrefs-exporter\")), Static(\"telemetry.sdk.version\"): String(Static(\"0.29.0\"))}, schema_url: None } }), readers: [PrometheusExporter { reader: ManualReader }], views: 0 }"
2025-05-21T09:16:47.586069Z DEBUG opentelemetry_sdk:  name="MeterProvider.Built"
2025-05-21T09:16:47.586078Z DEBUG opentelemetry_sdk:  name="MeterProvider.NewMeterCreated" meter_name="lustre"
2025-05-21T09:16:47.631614Z  INFO opentelemetry_sdk:  name="MeterProvider.Drop" Last reference of MeterProvider dropped, initiating shutdown.
2025-05-21T09:16:47.631627Z DEBUG opentelemetry_sdk:  name="MeterProvider.Drop.ShutdownCompleted"
2025-05-21T09:16:47.631635Z  WARN lustrefs_exporter: Content left in input buffer. Unparsed content: 'ost.OSS.ost_io.stats=
snapshot_time             17'. Please run and supply to support: `lctl get_param memused memused_max lnet_memused health_check mdt.*.exports.*.uuid osd-*.*.filesfree osd-*.*.filestotal osd-*.*.fstype osd-*.*.kbytesavail osd-*.*.kbytesfree osd-*.*.kbytestotal osd-*.*.brw_stats osd-*.*.quota_slave.acct_group osd-*.*.quota_slave.acct_user osd-*.*.quota_slave.acct_project mgs.*.mgs.stats mgs.*.mgs.threads_max mgs.*.mgs.threads_min mgs.*.mgs.threads_started mgs.*.num_exports obdfilter.*OST*.stats obdfilter.*OST*.num_exports obdfilter.*OST*.tot_dirty obdfilter.*OST*.tot_granted obdfilter.*OST*.tot_pending obdfilter.*OST*.exports.*.stats ost.OSS.ost.stats ost.OSS.ost_io.stats ost.OSS.ost_create.stats ost.OSS.ost_out.stats ost.OSS.ost_seq.stats mds.MDS.mdt.stats mds.MDS.mdt_fld.stats mds.MDS.mdt_io.stats mds.MDS.mdt_out.stats mds.MDS.mdt_readpage.stats mds.MDS.mdt_seqm.stats mds.MDS.mdt_seqs.stats mds.MDS.mdt_setattr.stats mdt.*.md_stats mdt.*MDT*.num_exports mdt.*MDT*.exports.*.stats ldlm.namespaces.{mdt-,filter-}*.contended_locks ldlm.namespaces.{mdt-,filter-}*.contention_seconds ldlm.namespaces.{mdt-,filter-}*.ctime_age_limit ldlm.namespaces.{mdt-,filter-}*.early_lock_cancel ldlm.namespaces.{mdt-,filter-}*.lock_count ldlm.namespaces.{mdt-,filter-}*.lock_timeouts ldlm.namespaces.{mdt-,filter-}*.lock_unused_count ldlm.namespaces.{mdt-,filter-}*.lru_max_age ldlm.namespaces.{mdt-,filter-}*.lru_size ldlm.namespaces.{mdt-,filter-}*.max_nolock_bytes ldlm.namespaces.{mdt-,filter-}*.max_parallel_ast ldlm.namespaces.{mdt-,filter-}*.resource_count ldlm.services.ldlm_canceld.stats ldlm.services.ldlm_cbd.stats llite.*.stats mdd.*.changelog_users qmt.*.*.glb-usr qmt.*.*.glb-prj qmt.*.*.glb-grp`
^C
[root@oss-server00 17:16 /tmp]# lctl get_param ost.OSS.ost_io.stats
ost.OSS.ost_io.stats=
snapshot_time             1747819110.946994856 secs.nsecs
start_time                1743990814.541172395 secs.nsecs
elapsed_time              3828296.405822461 secs.nsecs
req_waittime              904413900 samples [usec] 2 8040619 15430680576943 3779124587047592319
req_qdepth                904413900 samples [reqs] 0 108 11467546 77798756
req_active                904413900 samples [reqs] 1 63 20606558592 922743075676
req_timeout               904413900 samples [sec] 50 50 45220695000 2261034750000
reqbuf_avail              1806525580 samples [bufs] 60 64 115488402415 7383116548961
ost_read                  688351744 samples [usec] -10486153 6777393 32502485105514 3998506888724083082
ost_write                 216024436 samples [usec] 17 34908408 12300334665932 1914164938120496174
ost_punch                 37720 samples [usec] 14 553310 21144928 2685387318602

[RDruon]

Hi @yangliping, thank you for the report. Could you test #88 and see if it helps?

[yangliping]

Thank you. I'll test it later and inform you of the outcome. However, after reviewing Lustre's source code, I'm wondering if it would be more appropriate to change the counter type from u64 to s64. Could you please take this into consideration?

 149 struct lprocfs_counter {
 150         __s64   lc_count;
 151         __s64   lc_min;
 152         __s64   lc_max;
 153         __s64   lc_sum;
 154         __s64   lc_sumsquare;
 155 };

https://git.whamcloud.com/?p=fs/lustre-release.git;a=blob;f=lustre/include/lprocfs_status.h;h=56634f57b4d7126006dfcd587be260d4c6be7120;hb=refs/heads/master#l149

[yangliping]

The negative value is no longer present in my environment. I'm currently unable to test it on a real system.