-
Notifications
You must be signed in to change notification settings - Fork 124
Description
Describe the bug
In my application I wanted to upgrade to the latest java apm agent version available (to date 1.55.1). But this leads to problems with invalid json around histogram metrics and infinity values. Problem that occurs with histograms:
2025-11-27 09:24:10,260 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: {
"accepted": 329,
"errors": [
{
"message": "decode error: data read error: v2.metricsetRoot.Metricset: v2.metricset.Samples: v2.metricsetSampleValue.Values: []float64: readNumberAsString: invalid number, error found in #10 byte of ...|5105400.0,"Infinity"|..., bigger context ...|000.0,3843071682022823400.0,4227378850225105400.0,"Infinity"],"counts":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,|...",
"document": "{"metricset":{"timestamp":1764235442476000,"samples":{"process_uptime":{"value":390.378},"avg_statistics_data":{"value":0.0},"processing_times_last_5_mins.histogram":{"values":[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,16.0,21.0,26.0,31.0,36.0,41.0,46.0,51.0,56.0,64.0,85.0,106.0,127.0,148.0,169.0,190.0,211.0,232.0,256.0,341.0,426.0,511.0,596.0,681.0,766.0,851.0,936.0,1024.0,1365.0,1706.0,2047.0,2388.0,2729.0,3070.0,3411.0,3752.0,4096.0,5461.0,6826.0,8191.0,9556.0,10921.0,12286.0,13651.0,15016.0,16384.0,21845.0,27306.0,32767.0,38228.0,43689.0,49150.0,54611.0,60072.0,65536.0,87381.0,109226.0,131071.0,152916.0,174761.0,196606.0,218451.0,240296.0,262144.0,349525.0,436906.0,524287.0,611668.0,699049.0,786430.0,873811.0,961192.0,1048576.0,1398101.0,1747626.0,2097151.0,2446676.0,2796201.0,3145726.0,3495251.0,3844776.0,4194304.0,5592405.0,6990506.0,8388607.0,9786708.0,11184809.0,12582910.0,13981011.0,15379112.0,16777216.0,22369621.0,27962026.0,33554431.0,39146836.0,44739241.0,50331646.0,55924051.0,61516456.0,67108864.0,89478485.0,111848106.0,134217727.0,156587348.0,178956969.0,201326590.0,223696211.0,246065832.0,268435456.0,357913941.0,447392426.0,536870911.0,626349396.0,715827881.0,805306366.0,894784851.0,984263336.0,1073741824.0,1431655765.0,1789569706.0,2147483647.0,2505397588.0,2863311529.0,3221225470.0,3579139411.0,3937053352.0,4294967296.0,5726623061.0,7158278826.0,8589934591.0,10021590356.0,11453246121.0,12884901886.0,14316557651.0,15748213416.0,17179869184.0,22906492245.0,28633115306.0,34359738367.0,40086361428.0,45812984489.0,51539607550.0,57266230611.0,62992853672.0,68719476736.0,91625968981.0,114532461226.0,137438953471.0,160345445716.0,183251937961.0,206158430206.0,229064922451.0,251971414696.0,274877906944.0,366503875925.0,458129844906.0,549755813887.0,641381782868.0,733007751849.0,824633720830.0,916259689811.0,1007885658792.0,1099511627776.0,1466015503701.0,1832519379626.0,2199023255551.0,2565527131476.0,2932031007401.0,3298534883326.0,3665038759251.0,4031542635176.0,4398046511104.0,5864062014805.0,7330077518506.0,8796093022207.0,10262108525908.0,11728124029609.0,13194139533310.0,14660155037011.0,16126170540712.0,17592186044416.0,23456248059221.0,29320310074026.0,35184372088831.0,41048434103636.0,46912496118441.0,52776558133246.0,58640620148051.0,64504682162856.0,70368744177664.0,93824992236885.0,117281240296106.0,140737488355327.0,164193736414548.0,187649984473769.0,211106232532990.0,234562480592211.0,258018728651432.0,281474976710656.0,375299968947541.0,469124961184426.0,562949953421311.0,656774945658196.0,750599937895081.0,844424930131966.0,938249922368851.0,1032074914605736.0,1125899906842624.0,1501199875790165.0,1876499844737706.0,2251799813685247.0,2627099782632788.0,3002399751580329.0,3377699720527870.0,3752999689475411.0,4128299658422952.0,4503599627370496.0,6004799503160661.0,7505999378950826.0,9007199254740991.0,10508399130531156.0,12009599006321320.0,13510798882111486.0,15011998757901652.0,16513198633691816.0,18014398509481984.0,24019198012642644.0,30023997515803304.0,36028797018963970.0,4.2033596522124624E16,48038396025285290.0,54043195528445950.0,60047995031606610.0,66052794534767270.0,72057594037927940.0,96076792050570580.0,120095990063213230.0,144115188075855870.0,168134386088498530.0,192153584101141150.0,216172782113783800.0,240191980126426460.0,264211178139069100.0,288230376151711740.0,384307168202282300.0,480383960252852900.0,576460752303423500.0,672537544353994100.0,768614336404564600.0,864691128455135200.0,960767920505705900.0,1056844712556276400.0,1152921504606847000.0,1537228672809129200.0,1921535841011411700.0,2305843009213694000.0,2690150177415976400.0,3074457345618258400.0,3458764513820541000.0,3843071682022823400.0,4227378850225105400.0,"Infinity"],"counts":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"type":"histogram"},"processing_times_statistics_last_5_mins.histogram":{"values":[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,16.0,21.0,26.0,31.0,36.0,41.0,46.0,51.0,56.0,64.0,85.0,106.0,127.0,148.0,169.0,190.0,211.0,232.0,256.0,341.0,426.0,511.0,596.0,681.0,766.0,851.0,936.0,1024.0,1365.0,1706.0,2047.0,2388.0,2729.0,3070.0,3411.0,3752.0,4096.0,5461.0,6826.0,8191.0,9556.0,10921.0,12286.0,13651.0,15016.0,16384.0,21845.0,27306.0,32767.0,38228.0,43689.0,49150.0,54611.0,60072.0,65536.0,87381.0,109226.0,131071.0,152916.0,174761.0,196606.0,218451.0,240296.0,262144.0,349525.0,436906.0,524287.0,611668.0,699049.0,786430.0,873811.0,961192.0,1048576.0,1398101.0,1747626.0,2097151.0,2446676.0,2796201.0,3145726.0,3495251.0,3844776.0,4194304.0,5592405.0,6990506.0,8388607.0,9786708.0,11184809.0,12582910.0,13981011.0,15379112.0,16777216.0,22369621.0,27962026.0,33554431.0,39146836.0,44739241.0,50331646.0,55924051.0,61516456.0,67108864.0,89478485.0,111848106.0,134217727.0,156587348.0,178956969.0,201326590.0,223696211.0,246065832.0,268435456.0,357913941.0,447392426.0,536870911.0,626349396.0,715827881.0,805306366.0,894784851.0,984263336.0,1073741824.0,1431655765.0,1789569706.0,2147483647.0,2505397588.0,2863311529.0,3221225470.0,3579139411.0,3937053352.0,4294967296.0,5726623061.0,7158278826.0,8589934591.0,10021590356.0,11453246121.0,12884901886.0,14316557651.0,15748213416.0,17179869184.0,22906492245.0,28633115306.0,34359738367.0,40086361428.0,45812984489.0,51539607550.0,57266230611.0,62992853672.0,68719476736.0,91625968981.0,114532461226.0,137438953471.0,160345445716.0,183251937961.0,206158430206.0,229064922451.0,251971414696.0,274877906944.0,366503875925.0,458129844906.0,549755813887.0,641381782868.0,733007751849.0,824633720830.0,916259689811.0,1007885658792.0,1099511627776.0,1466015503701.0,1832519379626.0,2199023255551.0,2565527131476.0,2932031007401.0,3298534883326.0,3665038759251.0,4031542635176.0,4398046511104.0,5864062014805.0,7330077518506.0,8796093022207.0,10262108525908.0,11728124029609.0,13194139533310.0,14660155037011.0,16126170540712.0,17592186044416.0,23456248059221.0,29320310074026.0,35184372088831.0,41048434103636.0,46912496118441.0,52776558133246.0,58640620148051.0,64504682162856.0,70368744177664.0,93824992236885.0,117281240296106.0,140737488355327.0,164193736414548.0,187649984473769.0,211106232532990.0,234562480592211.0,258018728651432.0,281474976710656.0,375299968947541.0,469124961184426.0,562949953421311.0,656774945658196.0,750599937895081.0,844424930131966.0,938249922368851.0,1032074914605736.0,1125899906842624.0,1501199875790165.0,1876499844737706.0,2251799813685247.0,2627099782632788.0,3002399751580329.0,3377699720527870.0,3752999689475411.0,4128299658422952.0,4503599627370496.0,6004799503160661.0,7505999378950826.0,9007199254740991.0,10508399130531156.0,12009599006321320.0,13510798882111486.0,15011998757901652.0,16513198633691816.0,18014398509481984.0,24019198012642644.0,30023997515803304.0,36028797018963970.0,4.2033596522124624E16,48038396025285290.0,54043195528445950.0,60047995031606610.0,66052794534767270.0,72057594037927940.0,96076792050570580.0,120095990063213230.0,144115188075855870.0,168134386088498530.0,192153584101141150.0,216172782113783800.0,240191980126426460.0,264211178139069100.0,288230376151711740.0,384307168202282300.0,480383960252852900.0,576460752303423500.0,672537544353994100.0,768614336404564600.0,864691128455135200.0,960767920505705900.0,1056844712556276400.0,1152921504606847000.0,1537228672809129200.0,1921535841011411700.0,2305843009213694000.0,2690150177415976400.0,3074457345618258400.0,3458764513820541000.0,3843071682022823400.0,4227378850225105400.0,"Infinity"],"counts":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"type":"histogram"}}}}"
}
]
}
"Infinity" is coming from the “+Inf” bucket that Micrometer uses for timers/distribution summaries.
JSON does not allow Infinity / NaN as numeric values, and the APM Server’s decoder expects these to be valid float64. As soon as it sees "Infinity" it fails with:
v2.metricsetSampleValue.Values: []float64: readNumberAsString: invalid number
To Reproduce
- Use Java APM Agent version 1.55.1
- Have histogram percentile publishing enabled in your metric configuration. E.g. like this:
return DistributionSummary.builder(name)
.description(description)
.publishPercentileHistogram()
.publishPercentiles(0.0, 0.99) // min value, 99th percentile
.distributionStatisticExpiry(Duration.ofMinutes(minutesToMonitor))
.register(registry); - Then start your application and make sure your metric gets recorded.
- Error with invalid json will occur.
Expected behavior
No invalid json should be produced by apm agent.