Ship binaries for Dart

[robertbastian]

For Dart we need to ship precompiled binaries, as there is not Rust compiler available in the Dart toolchain (and no appetite to add one). However, the pub.dev package manager has a size limit of 100MB per package, which would be enough if we didn't have to ship separate binaries for all 27 supported platforms.

This means we will need some other mechanism for distributing binaries. Options include:

• GitHub releases
  • This should be the easiest way. We already have a proof-of-concept generating these using GitHub actions: https://github.com/robertbastian/icu4x/blob/728eb449584da57acc604466702a603c03a82e45/.github/workflows/artifacts-build.yml#L713-L759
• Unicode CDN
  • A Unicode CDN is a long term plan for ICU4X distribution on the web. It does not exist yet, and the idea was more to use it to serve data packages to users, not binaries to developers.

In the library, we add build.dart build script, which will download the binary from the source we're going with. Some open questions here are

• How do we configure this? There's currently no mechanism in Dart to parameterise dependencies
• Do we provide a non-network escape hatch, if the developer provides an ICU4X build themselves?
  • How do we handle versioning?

@mosuem

[mosuem]

Just a few notes: https://github.com/mosuem/generate_icu4x_binaries is where I am currently testing the binary generation (based on your proof of concept), which is then consumed by https://github.com/dart-lang/i18n/blob/17757fc7293fdbbd5e2942135e6867defb8496ec/pkgs/intl4x/build.dart.

For the open questions:

How do we configure this? There's currently no mechanism in Dart to parameterise dependencies

I am not sure I understand - the build.dart gets the target OS and architecture, and would only build/download for that target.

Do we provide a non-network escape hatch, if the developer provides an ICU4X build themselves?

Yes. Currently, the "escape hatch" is provided through the ICU4X_BUILD_MODE environment variable: Either

• local for a locally stored dylib
• checkout for a local checkout of the ICU4X repo to build a fresh dylib
• fetch to get the dylib from the CDN/GitHub releases.

There is an open issue on a better way of passing these parameters than env variables, possibly using Darts pubspec.yaml.

[robertbastian]

Ah yes that issue is what I meant.

[robertbastian]

Conclusion:

• In dart-lang/i18n we host:
  • per dev-platform dylib for dev (with nice errors and stuff)
  • per dev-platform data filtering tool
  • per target-platform static lib (highly optimized, stripped, panic-immediate-abort)
  • complete data file
• In the build.dart, we store fingerprints for these.
• The default behaviour of build.dart is to download. Environment variables can be used to use a local dylib and/or staticlib

LGTM: @Manishearth @mosuem @sffc @robertbastian

[robertbastian]

• @mosuem - checked with pub.dev team whether we could host artifacts there, they are reluctant. big discussion on the ecosystem team whether a package manager should support that. pub.dev team suggested github releases with fingerprinting. not great because packages are not self-contained anymore, but safe enough if we use fingerprinting
• @sffc - how do we get above 100MB?
• @robertbastian - we need 20+ platforms

...

cargo +nightly rustc --crate-type staticlib --profile=release-opt-size --no-default-features --features looping_panic_handler,libc_alloc,buffer_provider,icu_calendar,icu_collator,icu_datetime,icu_decimal,icu_list,icu_locid_transform,icu_plurals,icu_timezone  -p icu_capi -Zbuild-std=core,alloc -Zbuild-std-features=panic_immediate_abort --target=x86_64-unknown-linux-gnu --config=profile.release.strip=true

AI(@robertbastian): - put locale ops behind a feature in icu_capi

@ResourceIdentifier
setLocales(@NeedsToBeConst List<Locale> locales, {@NeedsToBeConst bool withFallback = true}) {
  // pragma:vm_never_remove
}