Create solana-sdk-wasm crate

[LucasSte]

Problem

solana-program and solana-sdk have both cdylib and rlib declared as crate types. Both crates are supposed to be used as libraries by other projects. This setting is specially problematic for SBF programs that import solana-program, since it does not permit us to run link-time optimizations:

   Compiling solana-program v2.2.1
error: lto can only be run for executables, cdylibs and static library outputs

error: could not compile `solana-program` (lib) due to 1 previous error

The problem lies in using cdylib and rlib together, since that prevents Rust from running LTO, when using Solana-program as a dependency. See rust-lang/rust#51009 for more information.

Summary of changes

1. Since the cdylib and the rlib types are only here to be compatible with the wasm target for wasm-pack, I created a new crate solana-sdk-wasm to hold all the items necessary for such a target.
2. Use only rlib for crate types in both solana-program and solana-sdk.
3. Updated the CI jobs accordingly.
4. Update the comments in program/src/lib.rs to correctly mention how to use the crate types.

Basic benchmark

This is not a comprehensive benchmark. I was just curious on the impact on a small Hello, Solana! program, like this:

entrypoint!(process_instruction);
fn process_instruction(
    _program_id: &Pubkey,
    accounts: &[AccountInfo],
    instruction_data: &[u8],
) -> ProgramResult {
    msg!("Hello, Solana!");
    Ok(())
}

These are the results:

No LTO: 18kb (18504 bytes) with 728 CUs consumed.
LTO enabled: 17kb (17576 bytes) with 638 CUs consumed.

[joncinque]

The main change introduced in this PR is that people need to run wasm-pack against your new package to get a binary that's usable in JS, and that should be the only breakage.

It's as if solana-sdk had a binary output and a library output (like solana-gossip), but we're just factoring the binary part into a new crate.

This means we can keep all of the code as is, but just re-export functions from the new crate, along with instructions for building. Does that make sense?

[LucasSte]

This means we can keep all of the code as is, but just re-export functions from the new crate, along with instructions for building. Does that make sense?

I didn't quite get the idea. Can I still maintain the wasm-specifc code in the new crate, but then re-export it in Solana-program and in solana-sdk? Or is it the opposite?

[joncinque]

I didn't quite get the idea. Can I still maintain the wasm-specifc code in the new crate, but then re-export it in Solana-program and in solana-sdk? Or is it the opposite?

Ah, sorry it wasn't clear. The wasm-specific code should still live in the original crate, and the new crate just re-exports everything and can be built into a JS package with wasm-pack.

[ilya-bobyr]

Problem

solana-program and solana-sdk have both cdylib and rlib declared as crate types. Both crates are supposed to be used as libraries by other projects. This crate setting is specially problematic for SBF programs that import solana-program, since it does not permit us to run link-time optimizations:

   Compiling solana-program v2.2.1
error: lto can only be run for executables, cdylibs and static library outputs

error: could not compile `solana-program` (lib) due to 1 previous error

Based on the above, it looks like in order to run LTO, you need the solana-program crate marked with cdylib or staticlib(?).
Though, it is unclear if this applies just to the final target type, or also to all the dependencies.
The error says "could not compile solana-program (lib)", so it seems that the dependencies need to be one of the specified types as well.

Summary of changes

1. Since the cdylib and the rlib types are only here to be compatible with the wasm target for wasm-pack, I created a new crate `solana-sdk-wasm` to hold all the items necessary for such a target.

2. Use only rlib for crate types in both `solana-program` and `solana-sdk`.

[...]

And yet, later, you are saying that rlib should be used for solana-program.
And it is in the diff, that lib.crate-type is changed from ["cdylib", "rlib"] to ["rlib"] (https://github.com/anza-xyz/solana-sdk/pull/138/files#diff-3c9d25a189ead32ebcc2ba81fb9b1a6ff81f14631141bbaf1202e0ddf98fe716R129-R131).

You are saying that you do see a size reduction. So, I'm probably missing something.
But I wonder if I am not the only one who might be confused here.

Or is it that the type of the dependent crates does not matter, and only the type of the final artifact matters?
If this is the case, then solana-program crate type should not be affecting the type of the crate that holds a specific program.
But you are narrowing the solana-program crate type.

[LucasSte]

@ilya-bobyr The problem is the usage of multiple crate types together. A short answer is that dependencies should be rlib and final objects should be cdylib, executables or staticlibs for LTO to run.

The option RUSTFLAGS= -C lto=fat is applied to all steps of the compilation, including the dependencies, so rustc will try to run LTO to generate the intermediate objects specified in the dependencies. In the case of Solana-program, rustc will generate an .so file for the crate, since it is specified in the crate types on its toml file. While rlib requires all functions to be present (no LTO needed), cdylib only requires the exported ones, so Rust doesn't work when you have two conflicting types of crates (this is a known issue in the compiler: rust-lang/rust#51009).

The plan is to support LTO via cargo-build-sbf, in which case we need the correct configuration of crate types. Since most solana programs depend on our own library, setting it to rlib facilitates it.

[LucasSte]

As per the discussion on Discord (https://discord.com/channels/428295358100013066/476811830145318912/1370445804078235908), I reverted the PR back to its original form. Let me know if there is any other feedback to be addressed.