Skip to content

Commit cf104b8

Browse files
samcamwilliamssamcamwilliams
authored
Merge pull request #20 from permaweb/feat/dev_process
Implement long-lived WASM exec in `hb_converge`, message caching during execution, and `dev_process` basics
2 parents 15e8fe9 + a394b97 commit cf104b8

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+5238
-2520
lines changed

‎.github/documentation/cd.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# CD
2+
3+
## Table of Contents
4+
- [Description](#description)
5+
- [Variables](#variables)
6+
- [Jobs](#jobs)
7+
- [Credentials](#credentials)
8+
9+
### Description
10+
This workflow is triggered when a push is made to the `main` branch and is responsible for building and deploying AO/HyperBEAM to a confidential VM in GCP.
11+
12+
### Variables
13+
14+
The following variables are defined by the workflow:
15+
- `GCP_PROJECT`: The GCP project to deploy the application to (hyperbeam-cd)
16+
- `GCP_IMAGE_NAME`: The name of the Packer image that is built (hyperbeam-image)
17+
- `GCP_INSTANCE_NAME`: The name of the GCP instance (hyperbeam)
18+
- `GCP_ZONE`: The GCP zone to deploy to (us-central1-a)
19+
20+
### Jobs
21+
22+
The workflow consists of four main jobs:
23+
24+
1. **build**:
25+
- Sets up the build environment with Erlang, Packer, and Rebar3
26+
- Builds and releases AO/HyperBEAM
27+
- Creates a Packer image with a unique name using timestamp and commit SHA
28+
- Tags the image with workflow run ID and commit SHA
29+
30+
2. **deploy**:
31+
- Creates a confidential AMD SEV-SNP VM using the built image
32+
- Configures the VM with secure boot, vTPM, and integrity monitoring
33+
34+
3. **test**:
35+
- Waits for deployment to complete
36+
- Runs tests (placeholder for actual test implementation)
37+
38+
4. **cleanup**:
39+
- Deletes the created VM instance
40+
- Cleans up old images, keeping only the last 5
41+
42+
### Credentials
43+
44+
The credentials are stored as a GitHub secret named `CD_SERVICE_ACCOUNT` containing GCP service account credentials.
45+
They were created as follows:
46+
```sh
47+
$ gcloud iam service-accounts create hyperbeam-cd-gha \
48+
--description="Service account for the hyperbeam-cd project" \
49+
--display-name="hyperbeam-cd-gha"
50+
```
51+
52+
and the `.json` credentials file was created as follows:
53+
```sh
54+
$ gcloud iam service-accounts keys create "hyperbeam-cd-gha.json" \
55+
--iam-account "[email protected]"
56+
```
57+
58+
The workflow uses these credentials to authenticate with Google Cloud using the `google-github-actions/auth` action.

‎.github/documentation/workflows.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Workflows
2+
3+
This document contains information about all of the workflows in the `.github/workflows` directory.
4+
These workflows are GitHub Actions workflows that are used for a variety of CI/CD tasks.
5+
6+
## Table of Contents
7+
8+
- [CD](cd.md)

‎.github/workflows/cd.yaml‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
name: Build and Deploy AO/HyperBEAM
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
8+
env:
9+
GCP_IMAGE_NAME: hyperbeam-image
10+
GCP_PROJECT: hyperbeam-cd
11+
GCP_INSTANCE_NAME: hyperbeam
12+
GCP_ZONE: us-central1-a
13+
14+
jobs:
15+
build:
16+
runs-on: ubuntu-latest
17+
outputs:
18+
image_name: ${{ steps.set_image_name.outputs.image_name }}
19+
steps:
20+
- uses: actions/checkout@v4
21+
- id: auth
22+
name: Authenticate to Google Cloud
23+
uses: google-github-actions/auth@v2
24+
with:
25+
credentials_json: ${{ secrets.CD_SERVICE_ACCOUNT }}
26+
27+
- name: Setup GCloud SDK
28+
uses: google-github-actions/setup-gcloud@v2
29+
30+
- name: Setup build tools (Erlang, Packer and Rebar3)
31+
run: |
32+
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
33+
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
34+
sudo apt-get update
35+
sudo apt-get install -y packer git libssl-dev ncurses-dev make cmake gcc g++
36+
git clone https://github.com/erlang/otp.git && cd otp && git checkout maint-27 && ./configure && make -j8 && sudo make install
37+
git clone https://github.com/erlang/rebar3.git && cd rebar3 && ./bootstrap && sudo mv rebar3 /usr/local/bin/
38+
39+
- name: Build and release AO/HyperBEAM with Rebar3
40+
run: |
41+
rebar3 clean
42+
rebar3 get-deps
43+
rebar3 compile
44+
rebar3 release
45+
46+
- name: Set image name with timestamp and commit SHA
47+
id: set_image_name
48+
run: |
49+
SHORT_SHA=$(echo ${{ github.sha }} | cut -c1-7)
50+
TIMESTAMPED_IMAGE_NAME="${{ env.GCP_IMAGE_NAME }}-${SHORT_SHA}-$(date +%Y%m%d-%H%M%S)"
51+
echo "image_name=${TIMESTAMPED_IMAGE_NAME}" >> "$GITHUB_OUTPUT"
52+
53+
- name: Build Packer Image
54+
run: |
55+
packer init .
56+
packer validate .
57+
packer build -var "image_name=${{ steps.set_image_name.outputs.image_name }}" -var "project_id=${{ env.GCP_PROJECT }}" .
58+
59+
- name: Tag image for reference
60+
run: |
61+
gcloud compute images add-labels ${{ steps.set_image_name.outputs.image_name }} \
62+
--project=${{ env.GCP_PROJECT }} \
63+
--labels=workflow_run=${{ github.run_id }},commit_sha=${{ github.sha }}
64+
65+
deploy:
66+
needs: build
67+
runs-on: ubuntu-latest
68+
steps:
69+
- id: auth
70+
name: Authenticate to Google Cloud
71+
uses: google-github-actions/auth@v2
72+
with:
73+
credentials_json: ${{ secrets.CD_SERVICE_ACCOUNT }}
74+
75+
- name: Setup GCloud SDK
76+
uses: google-github-actions/setup-gcloud@v2
77+
78+
- name: Create Confidential VM
79+
run: |
80+
gcloud compute instances create ${{ env.GCP_INSTANCE_NAME }} \
81+
--zone=${{ env.GCP_ZONE }} \
82+
--machine-type=n2d-standard-2 \
83+
--min-cpu-platform="AMD Milan" \
84+
--confidential-compute-type=SEV_SNP \
85+
--maintenance-policy=TERMINATE \
86+
--image-family=ubuntu-2404-lts-amd64 \
87+
--image-project=ubuntu-os-cloud \
88+
--project=${{ env.GCP_PROJECT }} \
89+
--network-interface=network-tier=PREMIUM,nic-type=GVNIC,stack-type=IPV4_ONLY,subnet=default \
90+
--tags=http-server,https-server \
91+
--shielded-secure-boot \
92+
--shielded-vtpm \
93+
--shielded-integrity-monitoring \
94+
--create-disk=auto-delete=yes,boot=yes,device-name=${{ env.GCP_INSTANCE_NAME }},image=projects/${{ env.GCP_PROJECT }}/global/images/${{ needs.build.outputs.image_name }},mode=rw,size=20,type=pd-balanced
95+
96+
test:
97+
needs: deploy
98+
runs-on: ubuntu-latest
99+
steps:
100+
- name: Wait for deployment
101+
run: sleep 60 # Add appropriate wait time for your service to start
102+
103+
- name: Run tests
104+
run: |
105+
# Add your test commands here
106+
echo "Running tests..."
107+
108+
cleanup:
109+
needs: [build, test] # Added build to needs to access the image name
110+
if: always()
111+
runs-on: ubuntu-latest
112+
steps:
113+
- id: auth
114+
name: Authenticate to Google Cloud
115+
uses: google-github-actions/auth@v2
116+
with:
117+
credentials_json: ${{ secrets.CD_SERVICE_ACCOUNT }}
118+
119+
- name: Setup GCloud SDK
120+
uses: google-github-actions/setup-gcloud@v2
121+
122+
- name: Delete Confidential VM
123+
run: |
124+
gcloud compute instances delete ${{ env.GCP_INSTANCE_NAME }} \
125+
--project=${{ env.GCP_PROJECT }} \
126+
--zone=${{ env.GCP_ZONE }} \
127+
--quiet
128+
129+
- name: Clean up old images
130+
run: |
131+
# Keep only the last 5 images
132+
gcloud compute images list \
133+
--project=${{ env.GCP_PROJECT }} \
134+
--filter="name ~ '^${{ env.GCP_IMAGE_NAME }}-'" \
135+
--format="get(name)" \
136+
--sort-by=~creationTimestamp \
137+
| tail -n +6 \
138+
| xargs -r gcloud compute images delete --quiet --project=${{ env.GCP_PROJECT }}

‎.gitignore‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,6 @@ node_modules
3030
c_src/*.o
3131
c_src/*.d
3232
priv/*
33-
rebar.lock
3433
.DS_STORE
3534
TEST-data*
3635
test-cache/*

‎erlang_ls.config‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,4 +24,4 @@ lenses:
2424
- show-behaviour-usages
2525
providers:
2626
enabled:
27-
- signature-help
27+
- signature-help

‎packer.pkr.hcl‎

Lines changed: 18 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ packer {
1010
# Define required variables
1111
variable "project_id" {
1212
type = string
13-
default = "arweave-437622"
13+
default = "hyperbeam-cd"
1414
}
1515

1616
variable "region" {
@@ -23,15 +23,16 @@ variable "zone" {
2323
default = "us-east1-c"
2424
}
2525

26-
variable "image_family" {
26+
variable "image_name" {
2727
type = string
28-
default = "ao-image"
28+
default = "hyperbeam-image"
2929
}
3030

3131
# Source block to define GCP builder
3232
source "googlecompute" "ubuntu" {
3333
project_id = var.project_id
3434
source_image_family = "ubuntu-2204-lts"
35+
image_name = var.image_name
3536
zone = var.zone
3637
machine_type = "n1-standard-1"
3738
ssh_username = "packer"
@@ -65,31 +66,31 @@ build {
6566
# Upload the pre-built release (with ERTS included) to the instance
6667
provisioner "file" {
6768
source = "./_build/default/rel/ao"
68-
destination = "/tmp/ao"
69+
destination = "/tmp/hyperbeam"
6970
}
7071

7172
provisioner "shell" {
7273
inline = [
7374
# Move the release to /opt with sudo
74-
"sudo mv /tmp/ao /opt/ao",
75-
"sudo chmod -R 755 /opt/ao",
75+
"sudo mv /tmp/hyperbeam /opt/hyperbeam",
76+
"sudo chmod -R 755 /opt/hyperbeam",
7677

7778
# Create a symlink to make it easier to run the app
78-
"sudo ln -s /opt/ao/bin/ao /usr/local/bin/ao",
79+
"sudo ln -s /opt/hyperbeam/bin/hyperbeam /usr/local/bin/hyperbeam",
7980

8081
# (Optional) If you want to create a systemd service to manage the app
81-
"echo '[Unit]' | sudo tee /etc/systemd/system/ao.service",
82-
"echo 'Description=Permaweb Node' | sudo tee -a /etc/systemd/system/ao.service",
83-
"echo '[Service]' | sudo tee -a /etc/systemd/system/ao.service",
84-
"echo 'Type=simple' | sudo tee -a /etc/systemd/system/ao.service",
85-
"echo 'ExecStart=/opt/ao/bin/ao foreground' | sudo tee -a /etc/systemd/system/ao.service",
86-
"echo 'Restart=on-failure' | sudo tee -a /etc/systemd/system/ao.service",
87-
"echo '[Install]' | sudo tee -a /etc/systemd/system/ao.service",
88-
"echo 'WantedBy=multi-user.target' | sudo tee -a /etc/systemd/system/ao.service",
82+
"echo '[Unit]' | sudo tee /etc/systemd/system/hyperbeam.service",
83+
"echo 'Description=Permaweb Node' | sudo tee -a /etc/systemd/system/hyperbeam.service",
84+
"echo '[Service]' | sudo tee -a /etc/systemd/system/hyperbeam.service",
85+
"echo 'Type=simple' | sudo tee -a /etc/systemd/system/hyperbeam.service",
86+
"echo 'ExecStart=/opt/hyperbeam/bin/hyperbeam foreground' | sudo tee -a /etc/systemd/system/hyperbeam.service",
87+
"echo 'Restart=on-failure' | sudo tee -a /etc/systemd/system/hyperbeam.service",
88+
"echo '[Install]' | sudo tee -a /etc/systemd/system/hyperbeam.service",
89+
"echo 'WantedBy=multi-user.target' | sudo tee -a /etc/systemd/system/hyperbeam.service",
8990

9091
# Enable and start the service
91-
"sudo systemctl enable ao",
92-
"sudo systemctl start ao"
92+
"sudo systemctl enable hyperbeam",
93+
"sudo systemctl start hyperbeam"
9394
]
9495
}
9596

‎rebar.config‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,8 @@
3333
{jiffy, {git, "https://github.com/ArweaveTeam/jiffy.git", {ref, "74c956defa9116c85d76f77c3e9b5bd6de7bd39a"}}},
3434
{cowboy, {git, "https://github.com/ninenines/cowboy", {tag, "2.12.0"}}},
3535
{prometheus, "4.11.0"},
36-
{prometheus_cowboy, "0.1.8"}
36+
{prometheus_cowboy, "0.1.8"},
37+
{rocksdb, "1.8.0"}
3738
]}.
3839

3940
{shell, [
@@ -51,3 +52,8 @@
5152
{include_erts, true},
5253
{extended_start_script, true}
5354
]}.
55+
56+
% {dist_node, [
57+
% {setcookie, 'hb'},
58+
% {name, 'hb@hb-node'}
59+
% ]}.

‎rebar.lock‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
{"1.2.0",
2+
[{<<"accept">>,{pkg,<<"accept">>,<<"0.3.5">>},2},
3+
{<<"b64fast">>,
4+
{git,"https://github.com/ArweaveTeam/b64fast.git",
5+
{ref,"58f0502e49bf73b29d95c6d02460d1fb8d2a5273"}},
6+
0},
7+
{<<"cowboy">>,
8+
{git,"https://github.com/ninenines/cowboy",
9+
{ref,"3ea8395eb8f53a57acb5d3c00b99c70296e7cdbd"}},
10+
0},
11+
{<<"cowlib">>,
12+
{git,"https://github.com/ninenines/cowlib",
13+
{ref,"1eb7f4293a652adcfe43b1835d22c58d8def839f"}},
14+
1},
15+
{<<"jiffy">>,
16+
{git,"https://github.com/ArweaveTeam/jiffy.git",
17+
{ref,"74c956defa9116c85d76f77c3e9b5bd6de7bd39a"}},
18+
0},
19+
{<<"prometheus">>,{pkg,<<"prometheus">>,<<"4.11.0">>},0},
20+
{<<"prometheus_cowboy">>,{pkg,<<"prometheus_cowboy">>,<<"0.1.8">>},0},
21+
{<<"prometheus_httpd">>,{pkg,<<"prometheus_httpd">>,<<"2.1.11">>},1},
22+
{<<"quantile_estimator">>,{pkg,<<"quantile_estimator">>,<<"0.2.1">>},1},
23+
{<<"ranch">>,
24+
{git,"https://github.com/ninenines/ranch",
25+
{ref,"a692f44567034dacf5efcaa24a24183788594eb7"}},
26+
1},
27+
{<<"rocksdb">>,{pkg,<<"rocksdb">>,<<"1.8.0">>},0}]}.
28+
[
29+
{pkg_hash,[
30+
{<<"accept">>, <<"B33B127ABCA7CC948BBE6CAA4C263369ABF1347CFA9D8E699C6D214660F10CD1">>},
31+
{<<"prometheus">>, <<"B95F8DE8530F541BD95951E18E355A840003672E5EDA4788C5FA6183406BA29A">>},
32+
{<<"prometheus_cowboy">>, <<"CFCE0BC7B668C5096639084FCD873826E6220EA714BF60A716F5BD080EF2A99C">>},
33+
{<<"prometheus_httpd">>, <<"F616ED9B85B536B195D94104063025A91F904A4CFC20255363F49A197D96C896">>},
34+
{<<"quantile_estimator">>, <<"EF50A361F11B5F26B5F16D0696E46A9E4661756492C981F7B2229EF42FF1CD15">>},
35+
{<<"rocksdb">>, <<"0AE072F9818DAC03E18BA0E4B436450D24040DFB1A526E2198B451FD9FA0284F">>}]},
36+
{pkg_hash_ext,[
37+
{<<"accept">>, <<"11B18C220BCC2EAB63B5470C038EF10EB6783BCB1FCDB11AA4137DEFA5AC1BB8">>},
38+
{<<"prometheus">>, <<"719862351AABF4DF7079B05DC085D2BBCBE3AC0AC3009E956671B1D5AB88247D">>},
39+
{<<"prometheus_cowboy">>, <<"BA286BECA9302618418892D37BCD5DC669A6CC001F4EB6D6AF85FF81F3F4F34C">>},
40+
{<<"prometheus_httpd">>, <<"0BBE831452CFDF9588538EB2F570B26F30C348ADAE5E95A7D87F35A5910BCF92">>},
41+
{<<"quantile_estimator">>, <<"282A8A323CA2A845C9E6F787D166348F776C1D4A41EDE63046D72D422E3DA946">>},
42+
{<<"rocksdb">>, <<"185E645EA480E9325D5EFE362BF3D2A38EDFC31B5145031B0CBEED978E89523C">>}]}
43+
].

0 commit comments

Comments
 (0)