Mainlining of Fast Snap logic (#804)

* supra: file tree_r ffi

* wire up treerfile to go

* supra: using the correct node reader in file tree_r ffi

* tree_r links

* redefs, it works

* missing test-supra cmd

* fast snap encode

* encode-snap test

* wire up fast-snap

* make gen

* encode works

* fix: yield in PSClientPoll correctly

* faster webui history

* pspoll: Don't timeout the context

* fix EpochPretty ddos

* more robust dealdata

* robusthttp metrics

* enable is done this way round

* pshare: prio snap

* encode bind to data

* make gen

* piecepark node dl limit

* only park pieces locally when supra P2 isn't running

* fix commp canaccept

* fix build

* fix commp canaccept storage query

* aaaargh

* prevent dupe piece park task

* expose piecepark min free pct

* Do snap falloc

* Cache storage info in localstorage.Local calls for a bit

* MoveStorage / UpgradeStore param tweaks

* snap key no read

* snap: Parallel movestorage

* snap mertics, ui fix

* missing snap metrics

* fix snap batch backoff

* somewhat handle snap msg fails on immutable deadline

* fix snap retry query params

* storage: Copy over snap vproof

* treeR file deadline

* slower retry movestorage

* ipni: Handle orphan tasks

* ipni: Accept orphans

* f05 bmgr

* f05 bmgr fixes

* fix f05 bm build

* make gen

* missing fixes postrebase

* fix: proofshare: Fix client payment chain bootstrap

* feat: test post cli: Vanilla Proof test command

* supra: live duration-so-far metric

* webui: Content page

* fix: Make kubo retrievals work (#724)

* fix(retrieval): handle pathed IPFS retrieval style, add basic retrieval tests (#720)

* ipni: Announce addresses with port

* update lotus

* make gen

---------

Co-authored-by: Rod Vagg <rod@vagg.org>

* add more caches

* more metrics, less log spam

* http method in req stats

* http metric for in-flight requests

* remotebs: return correct err on not-found

* limit parallel retrievals

* separate head req limiter

* fix race condition that resulted in ignoring the last bit of cached data.

* merge iffy

* gen

* lint and tests

* old linter making mistakes

* linty

* fixes

* oops

* agent review fixes

* cuda13 capable ffi

* supra: allow passing march flag

* Unified Snap (#806)

* build

* 2

* 3

* build gotchas

* right shape

* requirements include cuda

* get build fixed

* cuda for everyone

* add golang for ci

* stop deleting my stuff

* try with gcc flag

* auto-nvme-setup opt-out

* ci build fix attempt

* fix outdated base image

* gofmt, flag- no avx512bf16

* try cuda setup

* do like working job

* try without cuda

* try cude in install-deps

* cuda info in github env

* make spdk happy

* deps

* gcc12

* ubuntu-latest is not 2404 but 2004

* ci: missing deps, fix ubuntus

* supra: rm mbx wrapper, fix poseidon constants wait, native default arch

* review fixes

---------

Co-authored-by: Łukasz Magiera <magik6k@gmail.com>

* version and dep

* dl v1

* missing market miners onchange initial setup

* test supra system-info command

* supra: add missing pc2_hash_files template instantiations

* supra: correct -march in pc2.cu to make blst not segfault

* cache supraseal

* fix: storage: Make declare usably fast

* sectormetadata: Recheck partition presence

* debug expmgr

* expmgr: Never try to extend dead sectors

* use our runners

* Trying to fix CI

* fixing CI

* indexing: gracefully handle no-unsealed-copy pieces

* solve CI lint errors

* CI cleanups

* CI: export regex

* test_env_vars

* cgo fix

* paux in fallback flow

* gen

* cuda probe without nvcc

* env to disable snap supra

* fix generic crosscompile, explicit native target

* gcc 13 for supra

* supra docs

* address review

* fix itests

---------

Co-authored-by: Łukasz Magiera <magik6k@gmail.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: Rod Vagg <rod@vagg.org>

Author: 3 people

.github/actions/install-deps/action.yml

@@ -1,22 +1,90 @@
 name: 'Install Dependencies'
-description: 'Install common dependencies'
+description: 'Install common dependencies including CUDA for supraseal linking'
 
 runs:
   using: 'composite'
   steps:
     - name: Install dependencies
       run: |
-        sudo apt-get install -y curl ca-certificates gnupg ocl-icd-opencl-dev libhwloc-dev
+        sudo apt-get update
+        sudo apt-get install -y \
+          build-essential \
+          gcc-13 g++-13 \
+          pkg-config \
+          autoconf automake libtool \
+          curl wget git \
+          ca-certificates gnupg \
+          python3 python3-pip python3-dev python3-venv \
+          xxd nasm \
+          ocl-icd-opencl-dev libhwloc-dev \
+          libfuse3-dev uuid-dev libssl-dev libnuma-dev libaio-dev libarchive-dev libkeyutils-dev libncurses-dev \
+          libgmp-dev libconfig++-dev
       shell: bash
 
-    - name: Fetch all tags
-      run: git fetch --all
+    - name: Set up GCC 13 as default
+      run: |
+        sudo update-alternatives --remove-all gcc 2>/dev/null || true
+        sudo update-alternatives --remove-all g++ 2>/dev/null || true
+        sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 100
+        sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-13 100
+        echo "CC=gcc-13" >> $GITHUB_ENV
+        echo "CXX=g++-13" >> $GITHUB_ENV
+      shell: bash
+
+    - name: Install CUDA Toolkit
+      run: |
+        if [ -f "/usr/local/cuda/bin/nvcc" ]; then
+          echo "CUDA already installed"
+          exit 0
+        fi
+        wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
+        sudo dpkg -i cuda-keyring_1.1-1_all.deb
+        rm cuda-keyring_1.1-1_all.deb
+        sudo apt-get update
+        sudo apt-get -y install cuda-toolkit
+      shell: bash
+
+    - name: Set up CUDA environment
+      run: |
+        echo "/usr/local/cuda/bin" >> $GITHUB_PATH
+        echo "CUDA_HOME=/usr/local/cuda" >> $GITHUB_ENV
+        # Set up CUDA stubs for linking without GPU
+        STUBS="/usr/local/cuda/lib64/stubs"
+        if [ -d "$STUBS" ]; then
+          sudo ln -sf "$STUBS/libcuda.so" "$STUBS/libcuda.so.1" 2>/dev/null || true
+          echo "LD_LIBRARY_PATH=$STUBS:/usr/local/cuda/lib64" >> $GITHUB_ENV
+          echo "LIBRARY_PATH=$STUBS:/usr/local/cuda/lib64" >> $GITHUB_ENV
+        fi
+      shell: bash
+
+    # NOTE: CI sometimes flakes on large/full fetches (HTTP/2 early EOF).
+    # We only need tags (and only from origin) for versioning/metadata, so keep
+    # this fetch minimal and retry with HTTP/1.1.
+    - name: Fetch tags (resilient)
+      run: |
+        set -euo pipefail
+        git config --global http.version HTTP/1.1
+        git config --global http.postBuffer 524288000 || true
+
+        # Fetch only from origin (if present) and only tags; retry on transient network errors.
+        tries=0
+        until [ "$tries" -ge 4 ]; do
+          tries=$((tries+1))
+          if git fetch origin --tags --force --prune --no-recurse-submodules; then
+            exit 0
+          fi
+          echo "git fetch failed (attempt $tries/4), retrying in $((tries*5))s..." >&2
+          sleep $((tries*5))
+        done
+        echo "git fetch failed after 4 attempts" >&2
+        exit 1
       shell: bash
 
     - name: Sync submodules
       run: git submodule sync
       shell: bash
 
     - name: Update submodules
-      run: git submodule update --init
-      shell: bash
+      run: git submodule update --init --recursive
+      shell: bash
+

.github/actions/setup-go/action.yml

@@ -26,26 +26,39 @@ RUN go mod download
 FROM ghcr.io/filecoin-shipyard/lotus-containers:lotus-v1.34.0-rc2-devnet AS lotus-test
 
 # Stage 3: Build the final image
-FROM myoung34/github-runner:ubuntu-noble AS curio-github-runner
+FROM myoung34/github-runner:ubuntu-24.04 AS curio-github-runner
 
 # Copy Go dependencies from the builder stage
 COPY --from=builder /go/pkg /go/pkg
 COPY --from=builder /go/bin /go/bin
 COPY --from=lotus-test /usr/local/bin/lotus /usr/local/bin/
 
-RUN apt update && apt install -y \
+RUN apt-get update && apt-get install -y \
       build-essential \
-      bzr pkg-config \
+      gcc-13 g++-13 \
+      pkg-config \
+      autoconf automake libtool \
       clang \
-      curl \
-      git \
-      hwloc \
+      curl wget git \
+      ca-certificates gnupg \
+      python3 python3-pip python3-dev python3-venv \
+      xxd \
+      nasm \
+      hwloc libhwloc-dev \
       jq \
-      libhwloc-dev wget \
       mesa-opencl-icd \
       ocl-icd-opencl-dev \
-      gcc-12 g++-12 \
-      ca-certificates gnupg
+      libfuse3-dev uuid-dev libssl-dev libnuma-dev libaio-dev libarchive-dev libkeyutils-dev libncurses-dev \
+      libgmp-dev libconfig++-dev gcc-13 g++-13 \
+      ca-certificates gnupg \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+# Set up GCC 13 as default
+RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 100 \
+    && update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-13 100 \
+    && update-alternatives --set gcc /usr/bin/gcc-13 \
+    && update-alternatives --set g++ /usr/bin/g++-13
 
 ENV RUSTUP_HOME=/usr/local/rustup \
     CARGO_HOME=/usr/local/cargo \

.github/image/Dockerfile

@@ -4,13 +4,12 @@
 FROM ubuntu:24.04
 
 ENV DEBIAN_FRONTEND=noninteractive
-ENV CUDA_VERSION=13.0
-ENV GCC_VERSION=12
+ENV GCC_VERSION=13
 
 # Install system dependencies
 RUN apt-get update && apt-get install -y \
     build-essential \
-    gcc-12 g++-12 \
+    gcc-13 g++-13 \
     nasm \
     pkg-config \
     autoconf automake libtool \
@@ -30,11 +29,11 @@ RUN apt-get update && apt-get install -y \
     python3-venv \
     && rm -rf /var/lib/apt/lists/*
 
-# Set up GCC 12 as default
-RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 100 && \
-    update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 100 && \
-    update-alternatives --set gcc /usr/bin/gcc-12 && \
-    update-alternatives --set g++ /usr/bin/g++-12
+# Set up GCC 13 as default
+RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 100 && \
+    update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-13 100 && \
+    update-alternatives --set gcc /usr/bin/gcc-13 && \
+    update-alternatives --set g++ /usr/bin/g++-13
 
 # Install CUDA Toolkit from NVIDIA Repository
 # Source: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=24.04&target_type=deb_local
@@ -43,14 +42,22 @@ RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86
     rm cuda-keyring_1.1-1_all.deb && \
     apt-get update && \
     apt-get -y install cuda-toolkit && \
-    ln -sf /usr/local/cuda /usr/local/cuda-13.0 && \
     rm -rf /var/lib/apt/lists/*
 
-# Set up CUDA environment
+# Set up CUDA environment and verify installation
 ENV PATH=/usr/local/cuda/bin:${PATH}
 ENV CUDA_HOME=/usr/local/cuda
 ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64
 
+# Verify CUDA installation and version
+RUN nvcc --version && \
+    CUDA_VERSION=$(nvcc --version | grep "release" | sed -n 's/.*release \([0-9]*\)\.\([0-9]*\).*/\1/p') && \
+    echo "Installed CUDA version: $CUDA_VERSION" && \
+    if [ "$CUDA_VERSION" -lt 12 ]; then \
+        echo "ERROR: CUDA $CUDA_VERSION is too old. Need CUDA 12 or newer."; \
+        exit 1; \
+    fi
+
 # Set working directory
 WORKDIR /workspace
 

.github/workflows/Dockerfile.supraseal-test

@@ -75,7 +75,7 @@ If you prefer to test without Docker on Ubuntu 24.04:
 ```bash
 # Install dependencies (requires sudo)
 sudo apt-get update && sudo apt-get install -y \
-    build-essential gcc-12 g++-12 nasm pkg-config \
+    build-essential gcc-13 g++-13 nasm pkg-config \
     autoconf automake libtool libssl-dev libnuma-dev \
     uuid-dev libaio-dev libfuse3-dev libarchive-dev \
     libkeyutils-dev libncurses-dev python3 python3-pip \
@@ -84,9 +84,9 @@ sudo apt-get update && sudo apt-get install -y \
 # Install Python tools
 pip3 install --break-system-packages meson ninja pyelftools
 
-# Set GCC 12 as default
-sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 100
-sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 100
+# Set GCC 13 as default
+sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 100
+sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-13 100
 
 # Build
 cd extern/supraseal