Reduce native debug files upload and storage size

[codelimit]

Problem Statement

Gradle script uploads compiled native shared object files which contains required debug info. In our case the size of this object is ~2Gb. If each CI build upload debug info, server storage may have issues soon.

Solution Brainstorm

I am not sure if this possible, but how about objcopy --only-keep-debug to standalone file? In our case it helps to cut 200Mb.
The ideal option is to convert debug info into breakpad format. In our case 2GB turns into 200MB and I don't see any symbolication problems.

Maybe such options already exist?

Best regards!

[linear-code]

GRADLE-92

[supervacuus]

Sorry for the late reply, @codelimit. Just to check, I understood you correctly:

• You have a 2 GiB shared library that SAGP uploads as part of the automated native symbol upload task
• Extracting the debug info with objcopy --only-keep-debug manually shaves off 200MiB, leaving you with 1.8GiB of debug info still needed to upload
• Using dump_syms to get breakpad symbols reduces the uploaded data to 200MiB, and you have somehow verified that it is not a problem for your stack traces

Is that the correct interpretation?

I don't know how you verified it, but you might have seen that sentry-cli (which SAGP wraps) already supports both debug symbol variants as input.

So, my understanding is that you would like SAGP to offer more configuration for the native symbols upload task, so you can apply the conversion yourself before the upload happens and point the upload task there. Is that correct?

The current SAGP implementation uses the debug-files upload command of sentry-cli on the entire build output directory, which is hardcoded and defers file iteration to sentry-cli. To allow this, SAGP would need to expose the source directory, provide include/exclude globbing, or both. Something like:

sentry {
  nativeSymbols {
    sources.setFrom(...)
    include("**/*.sym")
  }
}

Independent of the required changes to SAGP, have you tried objcopy --only-keep-debug --compress-debug-sections=zlib yourlib.so yourlib.so.debug? It likely won't shave off as much as using breakpad symbol files, but at least you will have symbolication of inlined functions.

[codelimit]

Hi @supervacuus , thank you for response.

Is that the correct interpretation?

Yes. Some more info (click to expand):

symbols not upoaded:

converted to breakpad and uploaded manually:

auto uploaded by SAGP:

stripped and compressed as you suggested (`objcopy --only-keep-debug --compress-debug-sections=zlib`) and uploaded manually:

So, my understanding is that you would like SAGP to offer more configuration for the native symbols upload task, so you can apply the conversion yourself before the upload happens and point the upload task there. Is that correct?

Actually I would like SAGP to do it automatically , depending on some option:

sentry {
    uploadNativeSymbols = true
    autoUploadNativeSymbols = true
    symbolGeneratorType = "breakpad"
}

just like Crashlytics do: https://firebase.google.com/docs/crashlytics/troubleshooting#android-upgrade-to-gradle-plugin-v3

Independent of the required changes to SAGP, have you tried objcopy --only-keep-debug --compress-debug-sections=zlib yourlib.so yourlib.so.debug? It likely won't shave off as much as using breakpad symbol files, but at least you will have symbolication of inlined functions.

I checked, exported compressed debug info occupies ~420MiB. But it turned out that it misses unwind info. To solve this, I may add "-Wl,--compress-debug-sections=zlib" to my app's build script and let sentry upload mylib.so s it automatically.

But anyways, it would be great if SAGP can support different symbols generators.

[supervacuus]

So, my understanding is that you would like SAGP to offer more configuration for the native symbols upload task, so you can apply the conversion yourself before the upload happens and point the upload task there. Is that correct?

Actually I would like SAGP to do it automatically , depending on some option:

sentry {
    uploadNativeSymbols = true
    autoUploadNativeSymbols = true
    symbolGeneratorType = "breakpad"
}

just like Crashlytics do: https://firebase.google.com/docs/crashlytics/troubleshooting#android-upgrade-to-gradle-plugin-v3

Okay, thanks for the clarification. I'll inform the team to adapt the title accordingly.

I checked, exported compressed debug info occupies ~420MiB. But it turned out that it misses unwind info. To solve this, I may add "-Wl,--compress-debug-sections=zlib" to my app's build script and let sentry upload mylib.so s it automatically.

This is not really an issue on Android, since we upload the final stack traces from the device. The unwind info is only necessary for minidump events, which contain a stack snapshot, and the stack trace is constructed by walking in the backend. For symbolication alone, this should be sufficient.