Skip to content

Java APM agent produces invalid json with infinity values in histogram metrics #916

@performDanielHoefler

Description

@performDanielHoefler

Describe the bug
In my application I wanted to upgrade to the latest java apm agent version available (to date 1.55.1). But this leads to problems with invalid json around histogram metrics and infinity values. Problem that occurs with histograms:
2025-11-27 09:24:10,260 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: {
"accepted": 329,
"errors": [
{
"message": "decode error: data read error: v2.metricsetRoot.Metricset: v2.metricset.Samples: v2.metricsetSampleValue.Values: []float64: readNumberAsString: invalid number, error found in #10 byte of ...|5105400.0,"Infinity"|..., bigger context ...|000.0,3843071682022823400.0,4227378850225105400.0,"Infinity"],"counts":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,|...",
"document": "{"metricset":{"timestamp":1764235442476000,"samples":{"process_uptime":{"value":390.378},"avg_statistics_data":{"value":0.0},"processing_times_last_5_mins.histogram":{"values":[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,16.0,21.0,26.0,31.0,36.0,41.0,46.0,51.0,56.0,64.0,85.0,106.0,127.0,148.0,169.0,190.0,211.0,232.0,256.0,341.0,426.0,511.0,596.0,681.0,766.0,851.0,936.0,1024.0,1365.0,1706.0,2047.0,2388.0,2729.0,3070.0,3411.0,3752.0,4096.0,5461.0,6826.0,8191.0,9556.0,10921.0,12286.0,13651.0,15016.0,16384.0,21845.0,27306.0,32767.0,38228.0,43689.0,49150.0,54611.0,60072.0,65536.0,87381.0,109226.0,131071.0,152916.0,174761.0,196606.0,218451.0,240296.0,262144.0,349525.0,436906.0,524287.0,611668.0,699049.0,786430.0,873811.0,961192.0,1048576.0,1398101.0,1747626.0,2097151.0,2446676.0,2796201.0,3145726.0,3495251.0,3844776.0,4194304.0,5592405.0,6990506.0,8388607.0,9786708.0,11184809.0,12582910.0,13981011.0,15379112.0,16777216.0,22369621.0,27962026.0,33554431.0,39146836.0,44739241.0,50331646.0,55924051.0,61516456.0,67108864.0,89478485.0,111848106.0,134217727.0,156587348.0,178956969.0,201326590.0,223696211.0,246065832.0,268435456.0,357913941.0,447392426.0,536870911.0,626349396.0,715827881.0,805306366.0,894784851.0,984263336.0,1073741824.0,1431655765.0,1789569706.0,2147483647.0,2505397588.0,2863311529.0,3221225470.0,3579139411.0,3937053352.0,4294967296.0,5726623061.0,7158278826.0,8589934591.0,10021590356.0,11453246121.0,12884901886.0,14316557651.0,15748213416.0,17179869184.0,22906492245.0,28633115306.0,34359738367.0,40086361428.0,45812984489.0,51539607550.0,57266230611.0,62992853672.0,68719476736.0,91625968981.0,114532461226.0,137438953471.0,160345445716.0,183251937961.0,206158430206.0,229064922451.0,251971414696.0,274877906944.0,366503875925.0,458129844906.0,549755813887.0,641381782868.0,733007751849.0,824633720830.0,916259689811.0,1007885658792.0,1099511627776.0,1466015503701.0,1832519379626.0,2199023255551.0,2565527131476.0,2932031007401.0,3298534883326.0,3665038759251.0,4031542635176.0,4398046511104.0,5864062014805.0,7330077518506.0,8796093022207.0,10262108525908.0,11728124029609.0,13194139533310.0,14660155037011.0,16126170540712.0,17592186044416.0,23456248059221.0,29320310074026.0,35184372088831.0,41048434103636.0,46912496118441.0,52776558133246.0,58640620148051.0,64504682162856.0,70368744177664.0,93824992236885.0,117281240296106.0,140737488355327.0,164193736414548.0,187649984473769.0,211106232532990.0,234562480592211.0,258018728651432.0,281474976710656.0,375299968947541.0,469124961184426.0,562949953421311.0,656774945658196.0,750599937895081.0,844424930131966.0,938249922368851.0,1032074914605736.0,1125899906842624.0,1501199875790165.0,1876499844737706.0,2251799813685247.0,2627099782632788.0,3002399751580329.0,3377699720527870.0,3752999689475411.0,4128299658422952.0,4503599627370496.0,6004799503160661.0,7505999378950826.0,9007199254740991.0,10508399130531156.0,12009599006321320.0,13510798882111486.0,15011998757901652.0,16513198633691816.0,18014398509481984.0,24019198012642644.0,30023997515803304.0,36028797018963970.0,4.2033596522124624E16,48038396025285290.0,54043195528445950.0,60047995031606610.0,66052794534767270.0,72057594037927940.0,96076792050570580.0,120095990063213230.0,144115188075855870.0,168134386088498530.0,192153584101141150.0,216172782113783800.0,240191980126426460.0,264211178139069100.0,288230376151711740.0,384307168202282300.0,480383960252852900.0,576460752303423500.0,672537544353994100.0,768614336404564600.0,864691128455135200.0,960767920505705900.0,1056844712556276400.0,1152921504606847000.0,1537228672809129200.0,1921535841011411700.0,2305843009213694000.0,2690150177415976400.0,3074457345618258400.0,3458764513820541000.0,3843071682022823400.0,4227378850225105400.0,"Infinity"],"counts":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"type":"histogram"},"processing_times_statistics_last_5_mins.histogram":{"values":[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,16.0,21.0,26.0,31.0,36.0,41.0,46.0,51.0,56.0,64.0,85.0,106.0,127.0,148.0,169.0,190.0,211.0,232.0,256.0,341.0,426.0,511.0,596.0,681.0,766.0,851.0,936.0,1024.0,1365.0,1706.0,2047.0,2388.0,2729.0,3070.0,3411.0,3752.0,4096.0,5461.0,6826.0,8191.0,9556.0,10921.0,12286.0,13651.0,15016.0,16384.0,21845.0,27306.0,32767.0,38228.0,43689.0,49150.0,54611.0,60072.0,65536.0,87381.0,109226.0,131071.0,152916.0,174761.0,196606.0,218451.0,240296.0,262144.0,349525.0,436906.0,524287.0,611668.0,699049.0,786430.0,873811.0,961192.0,1048576.0,1398101.0,1747626.0,2097151.0,2446676.0,2796201.0,3145726.0,3495251.0,3844776.0,4194304.0,5592405.0,6990506.0,8388607.0,9786708.0,11184809.0,12582910.0,13981011.0,15379112.0,16777216.0,22369621.0,27962026.0,33554431.0,39146836.0,44739241.0,50331646.0,55924051.0,61516456.0,67108864.0,89478485.0,111848106.0,134217727.0,156587348.0,178956969.0,201326590.0,223696211.0,246065832.0,268435456.0,357913941.0,447392426.0,536870911.0,626349396.0,715827881.0,805306366.0,894784851.0,984263336.0,1073741824.0,1431655765.0,1789569706.0,2147483647.0,2505397588.0,2863311529.0,3221225470.0,3579139411.0,3937053352.0,4294967296.0,5726623061.0,7158278826.0,8589934591.0,10021590356.0,11453246121.0,12884901886.0,14316557651.0,15748213416.0,17179869184.0,22906492245.0,28633115306.0,34359738367.0,40086361428.0,45812984489.0,51539607550.0,57266230611.0,62992853672.0,68719476736.0,91625968981.0,114532461226.0,137438953471.0,160345445716.0,183251937961.0,206158430206.0,229064922451.0,251971414696.0,274877906944.0,366503875925.0,458129844906.0,549755813887.0,641381782868.0,733007751849.0,824633720830.0,916259689811.0,1007885658792.0,1099511627776.0,1466015503701.0,1832519379626.0,2199023255551.0,2565527131476.0,2932031007401.0,3298534883326.0,3665038759251.0,4031542635176.0,4398046511104.0,5864062014805.0,7330077518506.0,8796093022207.0,10262108525908.0,11728124029609.0,13194139533310.0,14660155037011.0,16126170540712.0,17592186044416.0,23456248059221.0,29320310074026.0,35184372088831.0,41048434103636.0,46912496118441.0,52776558133246.0,58640620148051.0,64504682162856.0,70368744177664.0,93824992236885.0,117281240296106.0,140737488355327.0,164193736414548.0,187649984473769.0,211106232532990.0,234562480592211.0,258018728651432.0,281474976710656.0,375299968947541.0,469124961184426.0,562949953421311.0,656774945658196.0,750599937895081.0,844424930131966.0,938249922368851.0,1032074914605736.0,1125899906842624.0,1501199875790165.0,1876499844737706.0,2251799813685247.0,2627099782632788.0,3002399751580329.0,3377699720527870.0,3752999689475411.0,4128299658422952.0,4503599627370496.0,6004799503160661.0,7505999378950826.0,9007199254740991.0,10508399130531156.0,12009599006321320.0,13510798882111486.0,15011998757901652.0,16513198633691816.0,18014398509481984.0,24019198012642644.0,30023997515803304.0,36028797018963970.0,4.2033596522124624E16,48038396025285290.0,54043195528445950.0,60047995031606610.0,66052794534767270.0,72057594037927940.0,96076792050570580.0,120095990063213230.0,144115188075855870.0,168134386088498530.0,192153584101141150.0,216172782113783800.0,240191980126426460.0,264211178139069100.0,288230376151711740.0,384307168202282300.0,480383960252852900.0,576460752303423500.0,672537544353994100.0,768614336404564600.0,864691128455135200.0,960767920505705900.0,1056844712556276400.0,1152921504606847000.0,1537228672809129200.0,1921535841011411700.0,2305843009213694000.0,2690150177415976400.0,3074457345618258400.0,3458764513820541000.0,3843071682022823400.0,4227378850225105400.0,"Infinity"],"counts":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"type":"histogram"}}}}"
}
]
}
"Infinity" is coming from the “+Inf” bucket that Micrometer uses for timers/distribution summaries.

JSON does not allow Infinity / NaN as numeric values, and the APM Server’s decoder expects these to be valid float64. As soon as it sees "Infinity" it fails with:

v2.metricsetSampleValue.Values: []float64: readNumberAsString: invalid number

To Reproduce

  1. Use Java APM Agent version 1.55.1
  2. Have histogram percentile publishing enabled in your metric configuration. E.g. like this:
    return DistributionSummary.builder(name)
    .description(description)
    .publishPercentileHistogram()
    .publishPercentiles(0.0, 0.99) // min value, 99th percentile
    .distributionStatisticExpiry(Duration.ofMinutes(minutesToMonitor))
    .register(registry);
  3. Then start your application and make sure your metric gets recorded.
  4. Error with invalid json will occur.

Expected behavior
No invalid json should be produced by apm agent.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions