Nexilis Builder (nex-builder)

Internal Component: This tool is primarily intended for use within the Function build jobs as part of the larger Nexilis system.

Overview

Nexilis Builder is a command-line interface (CLI) tool written in Go, designed to automate the process of building WebAssembly (WASI) components from various source languages and publishing them as OCI artifacts.

Its main responsibilities are:

1. Pulling Source Code: Fetching project source code from an S3-compatible object storage bucket.
2. Building WASI Components: Compiling the fetched source code into a WASI component (.wasm file) using the appropriate build toolchain for the language (currently supporting Rust with cargo component).
3. Pushing OCI Artifacts: Packaging the compiled WASM component into an OCI image format (specifically using WASM media types) and pushing it to an OCI-compliant registry (like Google Artifact Registry, Docker Hub, etc.).

This tool is typically run within a containerized environment where it has access to the necessary build tools (Rust, cargo-component) and credentials.

Features

• S3 Source Fetching: Downloads source code archives or directories from S3 using provided credentials and paths.
• Extensible Building Backend: Modular architecture designed to support multiple language backends (currently Rust, with plans for others like TypeScript, C++, etc.).
• WASI Compilation: Currently executes cargo component build --release for Rust projects to produce optimized WASI components.
• OCI Artifact Packaging: Creates an OCI manifest using appropriate WASM media types (application/vnd.bytecodealliance.wasm.component.layer.v0+wasm).
• OCI Registry Push: Pushes the packaged WASM artifact to a specified OCI registry using credentials.
• Configuration: Flexible configuration via command-line flags and environment variables (managed by Viper).
• Structured Logging: Outputs JSON logs for easier parsing in automated environments.

Prerequisites

To build and run the nex-builder Docker image or binary:

• Go: Version 1.24 or later (to build the builder itself).
• Docker: Required to build and run the containerized version.
• Language Toolchains (within Docker image): The Dockerfile includes the necessary toolchains for supported languages:
  • Rust: Includes the Rust toolchain and cargo-component
  • Additional Languages: Future versions will include toolchains for other supported languages
• Credentials:
  • S3: Access Key ID and Secret Access Key for the source bucket.
  • OCI Registry: Credentials formatted in a Docker config.json file (.dockerconfigjson) for the target OCI registry.

Configuration

Configuration is primarily handled via environment variables or command-line flags. Flags take precedence over environment variables.

Common Flags/Variables:

• --debug / -d: Enable debug level logging (Default: false).

pull Command Configuration:

Flag	Environment Variable	Required	Default	Description
--endpoint	NEXILIS_S3_ENDPOINT	Yes		S3-compatible endpoint URL.
--access-key	NEXILIS_S3_ACCESS_KEY	Yes		S3 Access Key ID.
--secret-key	NEXILIS_S3_SECRET_KEY	Yes		S3 Secret Access Key.
--use-ssl	NEXILIS_S3_USE_SSL	No	false	Use HTTPS/SSL for S3 connection.
--bucket	NEXILIS_S3_BUCKET	Yes		S3 Bucket name.
--source-path	NEXILIS_S3_PATH	Yes		Path (prefix/key) within the bucket to download.
--destination-path	NEXILIS_LOCAL_DESTINATION_PATH	Yes	/src	Local filesystem path to download source code to.

build Command Configuration:

Flag	Environment Variable	Required	Default	Description
--source / -s	NEXILIS_BUILD_SOURCE_PATH	No	/src	Path to the source code directory (containing project files).
--creds-file / -c	NEXILIS_BUILD_CONFIG	No	/creds/registry.json	Path to the OCI registry credentials file (config.json).
--repo / -r	NEXILIS_BUILD_REPOSITORY	Yes		Target OCI repository (e.g., my-registry.com/my-project).
--fn-name		Yes		Function name, used for image naming within the repository.
--tag / -t		No	latest	Tag to apply to the built OCI artifact.

receive Command Configuration:

Flag	Environment Variable	Required	Default	Description
--port / -p	NEXILIS_RECEIVE_PORT	Yes		Network port for the server to listen on.
--destination-path	NEXILIS_RECEIVE_DESTINATION_PATH	Yes	/src	Local filesystem path where the received tarball will be unpacked.

Credentials Files:

• OCI Credentials (--creds-file): This file should be in the standard Docker config.json format. It typically contains authentication details for your target OCI registry.
• S3 Credentials: Provided directly via flags or environment variables.

Usage

The builder is intended to be run within its Docker container. The typical workflow involves two steps, often orchestrated in a CI/CD pipeline (like the provided Kubernetes Job example):

1. Pull Source Code: Run the pull command to download source code from S3 into a shared volume.
2. Build and Push: Run the build command, mounting the shared volume (containing the source code) and the OCI credentials file.

Alternatively, the receive command can be used to accept source code directly via HTTP:

1. Start Receiver: Run the receive command in one container/process.
2. Send Tarball: From another process or machine, send a tarball via HTTP POST.

Example Commands:

# Assuming running within the container environment

# 1. Pull source from S3 bucket 'my-functions' path 'src/hello-wasi' to /workspace/src
builder pull \
  --endpoint s3.example.com \
  --access-key "MY_ACCESS_KEY" \
  --secret-key "MY_SECRET_KEY" \
  --bucket my-functions \
  --source-path src/hello-wasi \
  --destination-path /workspace/src

# 2. Build the code in /workspace/src and push to registry 'my-registry.com/functions'
#    Requires /creds/registry.json to contain OCI credentials.
builder build \
  --source /workspace/src \
  --creds-file /creds/registry.json \
  --repo my-registry.com/functions \
  --fn-name hello-wasi \
  --tag v1.0.1

# --- OR ---

# 1. Start the receiver on port 9000, unpacking to /workspace/received
builder receive --port 9000 --destination-path /workspace/received

# 2. (From another terminal/machine) Send a tarball
# Create a sample tar: tar cf my_project.tar ./my_project_src/
# Send it:
curl -X POST --data-binary @my_project.tar http://<receiver-ip-or-hostname>:9000/
# Or a gzipped tar: tar czf my_project.tar.gz ./my_project_src/
# Send it:
curl -X POST --data-binary @my_project.tar.gz http://<receiver-ip-or-hostname>:9000/

Building the Builder

Building the Go Binary Locally

go build -o nex-builder ./...

Building the Docker Image

The Dockerfile defines a multi-stage build process that includes the Go build, Rust setup, and cargo-component installation.

# Ensure Docker buildkit is enabled (often default)
# export DOCKER_BUILDKIT=1

docker build -t nexilis/builder:latest .
# Or use a specific tag matching your registry/versioning scheme
docker build -t my-registry.com/nexilis/builder:v0.1.0 .

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.