Skip to content

Port history from agave/sdk #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2,747 commits into from
Feb 7, 2025
Merged

Port history from agave/sdk #1

merged 2,747 commits into from
Feb 7, 2025

Conversation

joncinque
Copy link
Collaborator

Problem

It's time to move over the solana-sdk into this repo, but it's not there!

Summary of changes

Following similar PRs such as solana-program/token#22, pull in the whole history of the Solana SDK into this repo. I ran the following commands:

# add the remote for the source repo
git remote add agave https://github.com/anza-xyz/agave.git
# fetch the master from spl under a different local branch
git fetch agave main:ag
git switch ag
# be safe, remove the remote
git remote rm agave
# filter the stake related files
python3 ../git-filter-repo \
  --path .github/workflows/sdk.yml \
  --path .github/actions/setup/action.yml \
  --path sdk/ \
  --path-rename .github/workflows/sdk.yml:.github/workflows/main.yml \
  --path-rename sdk/: \
  --force
# re-add this repo
git remote add origin [email protected]:joncinque/solana-sdk.git
git fetch origin master
git switch master
git switch ag
git rebase master

This PR will be rebased onto master to preserve history

LucasSte and others added 30 commits February 7, 2025 18:16
* extract solana-msg crate

* allow println! in sdk/msg/src/lib.rs

* fix doc example

* put dep behind [target.'cfg(target_os = "solana")'.dependencies]

* fix import(?)
* VoteState::deserialize_into: take &mut MaybeUninit<VoteState>

Deserializing into MaybeUninit<VoteState> saves the extra cost of
initializing into a value initialized with VoteState::default()
Based on previous Turbine fanout experiment, wider fanouts are more
effective in propagating shreds and reducing repairs:
https://discord.com/channels/428295358100013066/478692221441409024/1265782094211321897

In order to identify optimal fanout value, this commit extends the
experiment with wider fanout values.
* fix: remove expensive versioned epoch stakes clone

* Add custom partialeq impl for dcou
* SDK: Add `PodSlotHashes` API

* SDK: Add tests for `PodSlotHashes` API

* SDK: Deprecate `SlotHashesSysvar` API
* add cfg(feature = "frozen-abi") to build.rs

* only depend on rustc_version when frozen-abi feature is activated

* remove extraneous dirs that snuck in from another branch

* update perf/build.rs as it's different from the standard build script

* use symlink for svm/build.rs

* remove unused build dep rustc_version from wen-restart

* fmt Cargo.toml
* improve type safety in stake state module

* feedback

* fix new method
* fix docs format

* fix wrong indent

* fix wrong indent

* fix wrong indent

* fix wrong indent

* fix wrong indent

* fix wrong indent

* revert bigtable docs update
* possible arithmetic_side_effects

* feedback

* fix test expected message

* fmt
* docs: fix coption doc test

* Update sdk/program/src/program_option.rs

Co-authored-by: Brooks <[email protected]>

* Update sdk/program/src/program_option.rs

Co-authored-by: Brooks <[email protected]>

---------

Co-authored-by: Brooks <[email protected]>
The functions manually computed seconds/millis/micros/nanos from a
Duration. The functions look to have been written before Duration
natively supported methods to calculate these different units.

So, remove our manual integer math and defer to whatever
std::time::Duration does
These functions are from a time when Duration did not natively support
these operations. Duration has now supported these methods for some
time, so we should do away with our functions and encourage any callers
to use the methods on Duration directly instead
* vote: deprecate legacy vote instructions

* fix tests
* bump rust to 1.80

* bump nightly version to 2024-07-21

* bump rust stable to 1.80.1, nightly to 2024-08-08

* clippy: macro_metavars_in_unsafe

* fix unexpected tag

* run anchor downstream test with their master

* add no-entrypoint into workspace level lint

* use correct llvm path for coverage test
* feat: support committing fee-only transactions

* update and add new tests

* fix sbf test

* feedback

* fix new test

* test cleanup
refactor: remove redundant references in entrypoint! macro
…#2618)

* sdk: Add `deserialize_into` to entrypoint

#### Problem

The main entrypoint for Solana programs allocates a vector on the heap
and pushes AccountInfos to it. Allocation is expensive.

#### Summary of changes

Add a new version of `deserialize` called `deserialize_into`, which
expects a slice of `MaybeUninit<AccountInfo>`. The entrypoint can
allocate a maximum array of AccountInfos on the stack and then pass it
in. This new version of the entrypoint saves roughly 30 CUs per unique
account passed to the program.

In an earlier version, I had the new function return the array itself,
but this used slightly more CUs, and didn't work for an array with 64
elements. Let me know how it looks!

* Call instruction processor in non-inlined function

* Add test for max supported accounts in a transaction

* Refactor extracting account infos and instruction data

* Changes required from rebase

* Add clippy allow

* Add panic message if too many accounts provided

* Add `entrypoint_no_alloc!` and revert behavior in entrypoint!

* Use entrypoint_no_alloc! everywhere except noop

* Comment why noop program works the way it does

* Add limit in doc-string

* CHANGELOG: Add entry for entrypoint
* Bump platform tools version

* Remove unused struct
* Splits transfer authority and finalize into two instructions.

* Adds next-version-forwarding to finalization.

* Makes loader-v4 a program runtime v1 loader.
* Add new feature key

* Wrap existing code with new feature

* Extend test harness

* Make test fail

* Populate EpochRewards::total_rewards from PointValue

* Remove superfluous struct field

* Fixup tests
* bump curve25519-dalek from 3.2.1 to 4.1.3

* Update .github/scripts/downstream-project-spl-common.sh

Co-authored-by: samkim-crypto <[email protected]>

* Update .github/scripts/downstream-project-spl-common.sh

Co-authored-by: samkim-crypto <[email protected]>

* Update .github/scripts/downstream-project-spl-common.sh

Co-authored-by: samkim-crypto <[email protected]>

* Update Cargo.toml

Co-authored-by: samkim-crypto <[email protected]>

* remove opt level hack

* add comment for opt level

---------

Co-authored-by: samkim-crypto <[email protected]>
* extract clock crate

* update clock usage in solana-program

* fmt

* fmt after rebase

* update lock file after rebase

* fmt after rebase

* fmt after rebase

* fmt

* make serde optional in solana-clock

* fix description

Co-authored-by: Jon C <[email protected]>

* fix docs link

Co-authored-by: Jon C <[email protected]>

* fix accidental deletions from workspace members table

---------

Co-authored-by: Jon C <[email protected]>
Lichtso and others added 26 commits February 7, 2025 18:16
* Rekeys feature.

* Disables new deployments on loader-v3 when the feature enable_loader_v4 is enabled.

* Adjusts tests.

* Adjusts tests for write lock demotion.
* Fix ALT_BN128_MULTIPLICATION_INPUT_LEN constant

* Add feature

* Apply suggestions from code review

Co-authored-by: samkim-crypto <[email protected]>

* Add SIMD

---------

Co-authored-by: samkim-crypto <[email protected]>
* extract loader-v4-interface crate

* missing re-exports

* fmt toml
* quic: increase timeout and keep alive

* quic: add comment to QUIC_MAX_TIMEOUT

---------

Co-authored-by: Lijun Wang <[email protected]>
This reverts commit 63241e6768f4f573c816e24268f6ca83b88040f7.
* sdk: Move things that won't be in the sdk out

#### Problem

The Solana sdk will soon have a standalone repository, but the `sdk/`
directory in the agave repo contains some things that won't move over.

#### Summary of changes

Move the following and update paths:

* cargo-build-sbf
* cargo-test-sbf
* sdk/sbf
* gen-headers

* Regenerate headers with gen-headers

* Update package paths for cargo-build-sbf tests

* Update CI scripts to look at platform-tools-sdk dir

* Move docker-solana into top-level, update paths

* Fix path in cargo-build-sbf test
* add the small crates to solana_program::example_mocks

* extract example-mocks crate

* finish extracting crate

* typo
#### Problem

As part of the work to separate the sdk from the rest of the agave repo,
we want to create a separate Cargo workspace in the `sdk/` directory.
Unfortunately, the `sdk/` directory also contains the implementation of
the `solana-sdk` crate.

#### Summary of changes

Move all of the code in `solana-sdk` (`src`, `tests`, `benches`,
`README.md`, `Cargo.toml`, etc) to `sdk/sdk`. Then update all local
paths to look at `sdk/sdk/` instead of `sdk/`
Adds feature gate disable_new_loader_v3_deployments.
This reverts commit 0eedd0b4a222aaacbf291bfe721721d1610c4027.
Changes loader-v4 truncate to set_program_length.
* Adds deprecation warnings in SDK.

* Adds deprecation warnings in CLI.
…yz#21 (#4676)

cleanup activated feature: Removing unwanted rounding in fee calculation anza-xyz#21
* start replacing program::stake

* replace solana_sysvar::stake_history (doesn't work because of dependency issue)

* patch solana-sysvar-id (unsuccessful)

* Fixup lockfile to use patch correctly

* use stake interface git dep for now

* remove now-unused import

* update lock file

* remove now-unused import

* add solana-clock patch

* missing feature activation

* fix lock file for patches

* another lock file fix

* add patches to svm/examples

* fmt

* missing feature activation

* update lock file

* missing feature activation

* add patch for frozen-abi

* update digest

* use published crate now that 1.2 is out

* add crates to patch-crates.sh

* undo adding clock crate (was already there)

* add crates_map entry for solana-sysvar

* missing solana-stake-interface/borsh in sdk/program

* update lock files

* trigger ci

* temp change to git dep with correct program ID

* switch back to crates.io stake interface

* use small crates in cargo-build-sbf tests to get around patching issue

* try removing dev dep

* activate solana-stake-interface/serde when serde is activated in solana-sysvar

* trigger ci

* trigger ci

* trigger ci

---------

Co-authored-by: Jon C <[email protected]>
#### Problem

`solana-sdk` depends on `solana-bn254` and `solana-secp256k1-recover`,
but they currently live in the `curves/` subdirectory. When the sdk gets
moved to a new Cargo workspace and repository, it needs to have all of
its solana crate dependencies, which is not currently the case.

#### Summary of changes

Move `curves/bn254` and `curves/secp256k1-recover` into `sdk/`, and
update all paths as needed.
#### Problem

During the work to extricate the sdk from the rest of the monorepo in
#4685, there were a couple of issues with CI.

First, the `dummy-for-ci-check` feature doesn't exist in any sdk crate,
so the `clippy` and `check` steps were failing when trying to activate
the feature.

Next, the frozen-abi tests were failing because the sdk doesn't activate
`solana-packet`'s frozen-abi feature when its frozen-abi is active.

#### Summary of changes

Add a `dummy-for-ci-check` to `solana-sdk`, and correctly activate
`frozen-abi` on `solana-packet`.
* sdk: Create new Cargo workspace for `sdk/`

#### Problem

We want to move everything under `sdk/` into a standalone repository,
but all of the crates are currently part of the monorepo workspace,
which makes the extraction impossible.

#### Summary of changes

Remove all crates in `sdk/` from the top-level workspace, but keep the
local dependency. Create a new `Cargo.toml` for the sdk workspace along
with all required dependencies and patches.

* Remove unused dependencies

* Update lockfile

* CI: Add sdk scripts to run through GitHub Actions

#### Problem

The monorepo contains a lot of testing steps for all of the packages.
Once the sdk lives in its own repos, it will need to run adequate CI,
just as before.

#### Summary of changes

Add all of the steps run over the sdk in the monorepo as scripts in
`sdk/scripts`, along with a setup step and workflow file.

These steps include:

* checking crate ownership (check-crates.sh)
* running hack checks (check-hack.sh)
* cargo check with nightly toolchain (check-nightly.sh)
* clippy (check-clippy.sh)
* crate sorting in Cargo.toml (check-sort.sh)
* dcou checks (check-dev-context-only-utils.sh)
* rustfmt check (check-fmt.sh)
* auditing crates (check-audit.sh)
* miri tests (test-miri.sh)
* frozen-abi tests (test-frozen-abi.sh)
* tests with stable toolchain (test-stable.sh)
* wasm tests (test-wasm.sh)
* coverage tests (test-coverage.sh)
* shellcheck (check-shell.sh)
* sanity checks (check-porcelain.sh and check-nits.sh)
* publish order (order-crates-for-publishing.py)
* benches (test-bench.sh)

A few differences with the monorepo testing:

* coverage reports are not uploaded
* bench results are not uploaded
* version check is omitted, since the sdk will version crates separately
* ssh key check is omitted, since there's no net scripts

Future work:

* downstream tests into solana-program repos
* semver checks
* spellchecking

* CI: Remove sdk specific commands and tests
…o.toml files (#4790)

* prevent check-crates from choking on PRs which do not touch any cargo.toml files in sdk/scripts/check-crates.sh
---------

Co-authored-by: Jon C <[email protected]>
Once chained Merkle shreds are rolled out, we should no longer use the
unchained variant.
* build(deps): bump bytes from 1.9.0 to 1.10.0

Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.9.0 to 1.10.0.
- [Release notes](https://github.com/tokio-rs/bytes/releases)
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md)
- [Commits](tokio-rs/bytes@v1.9.0...v1.10.0)

---
updated-dependencies:
- dependency-name: bytes
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update all Cargo files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* no resize area for unaligned programs
@joncinque joncinque requested a review from willhickey February 7, 2025 17:19
@joncinque joncinque merged commit 610df6e into anza-xyz:master Feb 7, 2025
@joncinque joncinque deleted the ag branch February 7, 2025 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.