Shorten the Ubuntu-ARM build time

[jmacd]

Pre-filing checklist

• I searched existing issues and didn't find a duplicate

Component(s)

CI / Build

Description

The ARM build time is over 50 minutes. It's tying up our merge queue.

Additional Context

See how I addressed this for Windows in https://github.com/open-telemetry/otel-arrow/pull/2695/changes#diff-a85a9e0c508de8a571c10d27dfcf1e903075cef27fd4d7216affeb90cabf8978R462.

It may help to look in to the reason why the build is so slow. For Windows I was able to confirm this using cargo build --timings directly; not sure for an ARM build.

Part of #2591