Inlined frames are skipped

[bobrik]

Describe the bug

During symbolication inlined frames are skipped, resulting in confusing profiles.

There's lots of inlining in real world software (especially with LTO).

To Reproduce

Compile this:

#include <stdio.h>

int innie() {
  for (;;) {
    printf(".");
  }

  return 0;
}

int middle() {
  return innie();
}

int __attribute__ ((noinline)) outie() {
  return middle();
}

int main() {
  return outie();
}

And run it on a host covered by Grafana Alloy with pyroscope.ebpf enabled.

Observe profiles missing middle:

If you try running addr2line manually with inlines requested, you'd see this:

ivan@cube:~$ llvm-addr2line -ifa -e derp 0x1164
0x1164
middle
/home/ivan/./derp.c:12
outie
/home/ivan/./derp.c:16

Expected behavior

Inline frames are present in profiles.

Additional Context

You can see that multiple frames can be returned, but only one is used:

https://github.com/grafana/opentelemetry-ebpf-profiler/blob/f910fc4db23e6552ed79507de3788f660a9471bc/pyroscope/symb/irsymcache/table.go#L45-L49

[bobrik]

You can see that multiple frames can be returned, but only one is used.

While they can be returned, only one is in fact returned, so taking the first one doesn't matter.

The problem is deeper in the stack.