Skip to content

Commit 10e2abc

Browse files
csasarakcsasarak
authored
Linux arm build (#1465)
* Temporary cancel build step while I figure out some stuff. * Empty container? * Confirm JS actions are disallowed. * They are disallowed. * Run single step in container. * Was it really running IN the container? * Run script. * need to get the path right. * Basic cargo builds with a target and setup. * Setup cargo target. * Check if we're on Linux correctly. * Fix toolchain * Is jemalloc the culprit? * Do I need build-essential? * Forget target for now, try build. * cache key calculation * Syntax fix. * Correct cache key conditional * Use build.sh for everything. * Fix conditional * Make cache-key computable * set -x and address some shellcheck issues. * Update vendor_download.sh to download arm assets. * Guard script arguments. Remove some old commentary. * My guards worked too well... * Run git config update just before build. * Change release verification to include arm * Only unit-tests. * Spec fix for building with arm. * aarch target. * hail mary rust build * One more target. * Does installing musl-dev work? * use sudo? * Try moving rust build into script. * More scripts. * syntax error. * Try with cc arg * revert me * Corrections. * env vars * Revert "Try with cc arg" This reverts commit 7c92287. * Try cross. * Revert "Try cross." This reverts commit fb866a8. * try using build-arm.sh * Non-interactive rustup. * Enable the environment. * Switch to lib-snippets. Configure link flags for Cargo. * path fix I hope * Propagate project files correctly. * conditionalize extra-source-files. * Print some env info. * Always save the cabal cache. * Commentary. * Fix cargo test. more debug output * temporarily disable cabal install check. * Help embeddedbinaries be findable when cabal is not running in root dir. * Temporarily disable diagnose. * Bring back cabal install check. * Revert "Bring back cabal install check." This reverts commit c56b7ce. * Cache key perms. * Permissions. * Try cleaning up. * Print home where cabal stuff goes. * echo paths * store-dir * Revert "store-dir" This reverts commit 2379f28. * Logic for LinuxARM cabal store. * MOre cabal store junk * Dummy cache test. * Try adding an additional rust-cache directory. * Re-enable other matrix builds. * Restore integration tests. * Update create release. * Try putting the cabal install check back. * Remove extra ls * Revert "Try putting the cabal install check back." This reverts commit 1e34a3b. * Bring back non linux arm cargo tests and diagnose checks. * Install nextest. * Don't need extra tooling on linux arm * Fix condition. * Cabal install check condition. * Cache OS. * Sha256 sum everything. * A little cleanup. * Fix quoting. * Insert arm generated bundles. * Add support for linux/arm64 to install-latest.sh * Only shasum fossa binaries. * Fix arch_version check so it doesn't trigger on macos. * Attempt to re-enable jemalloc on linux x86_64. * Remove dead code. * Revert conditional extra-source-files. * Final cleanup. * Update changelog. * jemalloc linux arm? * Better integrate jemalloc on arm. * Fix duplicated tests. * Use /runner/. $HOME is not set correctly. * Fix script oops.
1 parent b1b49c7 commit 10e2abc

File tree

13 files changed

+263
-107
lines changed

13 files changed

+263
-107
lines changed

‎.cargo/config.toml

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[target.aarch64-unknown-linux-musl]
2+
rustflags = ["-C", "link-arg=-lgcc"]

‎.github/workflows/build-all.yml

+109-92
Large diffs are not rendered by default.

‎.github/workflows/scripts/build-arm.sh

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#!/usr/bin/env sh
2+
3+
set -e
4+
set -x
5+
6+
if [ $# -lt 2 ] ; then
7+
echo "Usage: ./build-arm.sh <RUNNER OS> <PROJECT FILE> [FEATURES]"
8+
exit 1
9+
fi
10+
11+
RUNNER_OS=$1
12+
PROJECT_FILE=$2
13+
FEATURES=$3
14+
15+
# Install rust tooling
16+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
17+
. "/home/runner/.cargo/env"
18+
19+
rustc -V
20+
cargo -V
21+
22+
# Build Rust binaries
23+
cargo build ${FEATURES:+--features $FEATURES} --release
24+
25+
cargo test --release
26+
27+
# Validate that diagnose runs
28+
cargo run --release --bin diagnose -- walk --trace-spans none --trace-level info
29+
30+
# Now do Haskell things
31+
./.github/workflows/scripts/build.sh "$RUNNER_OS" "$PROJECT_FILE"

‎.github/workflows/scripts/build.sh

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
#!/usr/bin/env sh
2+
3+
set -e
4+
set -x
5+
6+
if [ $# -lt 2 ] ; then
7+
echo "Usage: ./build.sh <RUNNER OS> <PROJECT FILE>"
8+
exit 1
9+
fi
10+
11+
RUNNER_OS=$1
12+
PROJECT_FILE=$2
13+
14+
make build-test-data
15+
16+
# Does this need to run in the container?
17+
# It normally runs outside.
18+
git config --global --add safe.directory "$GITHUB_WORKSPACE"
19+
20+
# With dist-newstyle caches:
21+
# Cabal mainly knows to recompile based on changes to files.
22+
# Tagging in git doesn't reliably change a file in a fixed location that cabal/GHC can track to indicate that there's a new tag.
23+
# For our release process, we merge to master, which builds (and may store a dist-newstyle cache), then push a release tag.
24+
# During the tag build, cabal/GHC may not realize that they have to rebuild the Version.hs file because the tag is invisible to it.
25+
# This line adds a comment to our version source file to prompt cabal/GHC to rebuild Version.hs.
26+
echo "{- $GITHUB_RUN_ID -}" >> src/App/Version.hs
27+
cabal update
28+
cabal build --project-file="$PROJECT_FILE" all
29+
cabal test --project-file="$PROJECT_FILE" unit-tests
30+
31+
# Todo: Bring this back.
32+
# Test cabal-install.
33+
# This check ensures that QuickImport can use spectrometer as a library.
34+
# if [ "$RUNNER_OS" = 'Linux' ] ; then
35+
# cabal install --overwrite-policy=always --project-file="$PROJECT_FILE" --ghc-options="-Wwarn"
36+
# fi

‎.github/workflows/scripts/compute_cache_key.sh

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#!/usr/bin/env sh
2+
3+
set -e
4+
set -x
5+
6+
if [ $# -lt 2 ] ; then
7+
echo "Usage: ./compute_cache_key <RUNNER OS> <PROJECT FILE>"
8+
exit 1
9+
fi
10+
11+
RUNNER_OS=$1
12+
PROJECT_FILE=$2
13+
14+
cabal --project-file="$PROJECT_FILE" update
15+
cabal --project-file="$PROJECT_FILE" build --dry-run
16+
jq '."install-plan"[]."id"' < dist-newstyle/cache/plan.json | sort > /tmp/cabal-cache-key
17+
echo "Install plan:"
18+
cat /tmp/cabal-cache-key
19+
20+
if [ "$RUNNER_OS" = "macOS" ]; then
21+
PLAN_SUM=$(shasum -a256 /tmp/cabal-cache-key)
22+
else
23+
PLAN_SUM=$(sha256sum /tmp/cabal-cache-key)
24+
fi
25+
26+
CABAL_CACHE_KEY="$(echo "$PLAN_SUM" | awk '{print $1}')"
27+
export CABAL_CACHE_KEY
28+
echo "Cabal cache key: $CABAL_CACHE_KEY"
29+
echo "cabal-cache-key=$CABAL_CACHE_KEY" >> "$GITHUB_OUTPUT"
30+
31+
echo "Home dotfiles in container: $(ls -al ~/.*)"
32+
33+
# Cleanup. Restoring this cache seems to fail if the directory already exists.
34+
rm -rf dist-newstyle

‎Cargo.lock

+1-1
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml

+4
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,7 @@ resolver = "2"
1717
[profile.release]
1818
lto = "fat"
1919
strip = true
20+
21+
# There is additional configuration in .cargo/config.toml.
22+
# It is used for building on aarch64-unknown-linux-musl.
23+
# More context on the split here: https://github.com/rust-lang/cargo/issues/12738

‎Changelog.md

+4
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
# FOSSA CLI Changelog
22

3+
## 3.9.32
4+
5+
- Platform Support: Add a binary for ARM64 Linux environments. ([#1465](https://github.com/fossas/fossa-cli/pull/1465))
6+
37
## 3.9.31
48

59
- Resolve an issue parsing toml configuration files. ([#1459](https://github.com/fossas/fossa-cli/pull/1459))

‎extlib/millhone/Cargo.toml

+1-1
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ tikv-jemallocator = { version = "0.5.4", optional = true }
1212
clap = { version = "4.3.21", features = ["derive", "env", "cargo"] }
1313
stable-eyre = "0.2.2"
1414
srclib = { version = "*", git = "https://github.com/fossas/foundation-libs" }
15-
snippets = { version = "0.1.3", git = "https://github.com/fossas/foundation-libs", features = ["lang-all"] }
15+
snippets = { version = "0.1.3", tag = "v0.1.3", git = "https://github.com/fossas/lib-snippets", features = ["lang-all"] }
1616
traceconf = { git = "https://github.com/fossas/foundation-libs", version = "1.1.0" }
1717
serde = { version = "1.0.183", features = ["derive"] }
1818
thiserror = "1.0.46"

‎install-latest.sh

+12-1
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ is_supported_platform() {
6666
darwin/amd64) found=0 ;;
6767
darwin/arm64) found=0 ;;
6868
linux/amd64) found=0 ;;
69+
linux/arm64) found=0 ;;
6970
esac
7071
return $found
7172
}
@@ -119,7 +120,16 @@ adjust_arch() {
119120
# adjust archive name based on ARCH
120121
true
121122
}
122-
123+
arch_version_check() {
124+
# TODO: Make this version correct before merging.
125+
if [ "${OS}/${ARCH}" = "linux/arm64" ] && version_less_than "${VERSION}" '3.9.32'
126+
then
127+
echo "There is no linux/arm64 binary for version $VERSION."
128+
echo "Please select a version that is at least version 3.9.32"
129+
exit 1
130+
fi
131+
true
132+
}
123133
cat /dev/null <<EOF
124134
------------------------------------------------------------------------
125135
https://github.com/client9/shlib - portable posix shell functions
@@ -482,6 +492,7 @@ adjust_os
482492

483493
adjust_arch
484494

495+
arch_version_check
485496

486497
NAME=$(get_binary_name)
487498
TARBALL=${NAME}.${FORMAT}

‎src/Data/FileEmbed/Extra.hs

+10-4
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,17 @@
11
module Data.FileEmbed.Extra (
22
embedFile',
33
embedFileIfExists,
4-
) where
4+
)
5+
where
56

67
import Control.Exception (try)
78
import Control.Monad (when)
8-
import Data.FileEmbed (embedFile)
9+
import Data.FileEmbed (embedFile, makeRelativeToProject)
910
import Data.Foldable (traverse_)
1011
import Data.Maybe (isJust)
1112
import Language.Haskell.TH (reportWarning, runIO)
1213
import Language.Haskell.TH.Syntax (Exp (LitE), Lit (..), Q)
13-
import Path (parseRelFile)
14+
import Path (parseAbsFile)
1415
import Path.IO (doesFileExist)
1516
import System.Directory (getCurrentDirectory, listDirectory)
1617
import System.Environment (lookupEnv)
@@ -19,7 +20,12 @@ import System.FilePath (isRelative, splitPath, takeDirectory)
1920
embedFileIfExists :: FilePath -> Q Exp
2021
embedFileIfExists inputPath = do
2122
skipEmbedEnvVar <- runIO $ lookupEnv "FOSSA_SKIP_EMBED_FILE_IN_HLS"
22-
case (skipEmbedEnvVar, parseRelFile inputPath) of
23+
-- During some commands (like cabal install) the working directory may not
24+
-- be the same as the root of this project.
25+
-- makeRelativeToProject takes a relative path and roots it in the highest directory
26+
-- a .cabal file is found and returns an absolute path.
27+
inputPath' <- makeRelativeToProject inputPath
28+
case (skipEmbedEnvVar, parseAbsFile inputPath') of
2329
(Just _, _) -> do
2430
pure (LitE $ StringL "")
2531
(_, Just path) -> do

‎test/App/Fossa/PathDependencySpec.hs

+6-4
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import Control.Algebra (Has)
1111
import Control.Effect.FossaApiClient (FossaApiClientF (..), PackageRevision (PackageRevision))
1212
import Data.Flag (toFlag)
1313
import Data.Set qualified as Set
14+
import Data.String.Conversion (toText)
1415
import Data.Text (Text)
1516
import DepTypes (
1617
DepEnvironment (EnvTesting),
@@ -41,10 +42,11 @@ absPathOfSpec = describe "absPathOfSpec" $ do
4142
mkPathSpec cwd "../" isAbsDir
4243
mkPathSpec cwd "./" isAbsDir
4344

44-
mkPathSpec cwd "../fossa-cli" isAbsDir
45-
mkPathSpec cwd "../fossa-cli/" isAbsDir
46-
mkPathSpec cwd "../fossa-cli/test" isAbsDir
47-
mkPathSpec cwd "../fossa-cli/test/" isAbsDir
45+
let currentPathRel = "../" <> (toText . toFilePath . dirname $ cwd)
46+
mkPathSpec cwd currentPathRel isAbsDir
47+
mkPathSpec cwd (currentPathRel <> "/") isAbsDir
48+
mkPathSpec cwd (currentPathRel <> "/test") isAbsDir
49+
mkPathSpec cwd (currentPathRel <> "/test/") isAbsDir
4850

4951
mkPathSpec cwd "./test" isAbsDir
5052
mkPathSpec cwd "./test/" isAbsDir

‎vendor_download.sh

+13-4
Original file line numberDiff line numberDiff line change
@@ -51,11 +51,20 @@ case "$(uname -s)" in
5151
;;
5252

5353
Linux)
54-
ASSET_POSTFIX="linux"
55-
THEMIS_ASSET_POSTFIX="linux-amd64"
56-
LERNIE_ASSET_POSTFIX="x86_64-linux"
57-
;;
54+
case "$(uname -m)" in
55+
aarch64)
56+
ASSET_POSTFIX="linux"
57+
THEMIS_ASSET_POSTFIX="linux-arm64"
58+
LERNIE_ASSET_POSTFIX="aarch64-linux"
59+
;;
5860

61+
*)
62+
ASSET_POSTFIX="linux"
63+
THEMIS_ASSET_POSTFIX="linux-amd64"
64+
LERNIE_ASSET_POSTFIX="x86_64-linux"
65+
;;
66+
esac
67+
;;
5968
*)
6069
echo "Warn: Assuming $(uname -s) is Windows"
6170
ASSET_POSTFIX="windows.exe"

0 commit comments

Comments
 (0)