Skip to content

Commit 48ea079

Browse files
mratsimpetarvujovic98smtmfft
authored
feat(task db): implement a task DB (#208)
* task-manager: dump fight vs sqlite * task-manager: SQL tables+views+triggers success - but arguments passed in execute are 'NULL' * task-manager: passing enqueue_task tests * task-manager: cleanup - ease copy pasting to SQL script, Registered status, persistent views, remove debug print * task-manager: add DB size query * task-manager: id_proof is unneeded + prettify queries * task-manager: change DB schema - allow multiple provers and status for same task in case of failures-retry * task-manager: allow task updates * task-manager: retrieve cached proofs from DB * task-manager: add status check * task-manager: add progress reports * chore(task_manager): Run cargo fmt * feat: address small lints * feat(task-manager): use result type with thiserror * chore(task-db): fix typos * refactor(task-manager): clean up tests * fix(docker): unignore task manager * [WIP](task_manager): write initial task handler stubs * chore(task_manager): run cargo fmt * [WIP](task_manager): write status and proof get handlers * refactor(host): use merge instead of nest * chore(format): format workflow files * chore(deps): use consistent dependency style * chore(host): rename tx to task_channel * [WIP](task_manager): add initial submit logic * chore(clippy): remove unused parameter * chore(clippy): remove unused imports * refactor(core): add copy trait to proof types * feat(task_manager): simplify db and adapt tests * fix(clippy): fix dereference issue * [WIP]: handle proof request by worker and update task status * [WIP]: add block fetching and initial blockhash getting for submit * [WIP]: handle task creation, status and proof retrieval * fix(host): fix route sub-path * feat(raiko): abstract task manager and impl a mem db for easy integration (#296) * impl a mem db for easy integration Signed-off-by: smtmfft <[email protected]> * fix clippy and unit test Signed-off-by: smtmfft <[email protected]> * fix fmt Signed-off-by: smtmfft <[email protected]> --------- Signed-off-by: smtmfft <[email protected]> * fix: throw error instead of panicing on runtime checks * fix(core,task_manager): add custom ensure and require fns * feat(task_db): sqlite and in memory abstraction (#301) * enable sqlite db by feature Signed-off-by: smtmfft <[email protected]> * debug lifetime Signed-off-by: smtmfft <[email protected]> * resolve lifetime issue and make all tests pass Signed-off-by: smtmfft <[email protected]> * refactor(task_db): simplify structure for sqlite and use cached statements * feat(task_db): abstract task db implementation into wrapper * fix(task_db): add await to test call * fix(task_db): fix import declaration * fix(task_db): add async and mutable variables * fix(host): fix task manager usage * fix(task_db): fix test for async * Update Cargo.toml use in-mem as default. --------- Signed-off-by: smtmfft <[email protected]> Co-authored-by: smtmfft <[email protected]> Co-authored-by: smtmfft <[email protected]> * feat(task_manager): return empty list on key not found * feat(host,task_manager): add tracing and handle workers * feat(host): fix response structure * chore(clippy): remove unused imports * fix(ci): remove git merge added lines * fix(task_manager): add blob proof type field --------- Signed-off-by: smtmfft <[email protected]> Co-authored-by: Petar Vujović <[email protected]> Co-authored-by: smtmfft <[email protected]> Co-authored-by: smtmfft <[email protected]>
1 parent 1d69947 commit 48ea079

27 files changed

+2813
-80
lines changed

‎.dockerignore

+1
Original file line numberDiff line numberDiff line change
@@ -19,3 +19,4 @@
1919
!/provers/sgx/setup
2020
!/kzg_settings_raw.bin
2121
!/core
22+
!/task_manager

‎.github/workflows/ci-native.yml

+6-7
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,10 @@ name: CI - Native
22

33
on: workflow_call
44

5-
65
jobs:
7-
build-test-native:
8-
name: Build and test native
9-
uses: ./.github/workflows/ci-build-test-reusable.yml
10-
with:
11-
version_name: "native"
12-
version_toolchain: "nightly-2024-04-17"
6+
build-test-native:
7+
name: Build and test native
8+
uses: ./.github/workflows/ci-build-test-reusable.yml
9+
with:
10+
version_name: "native"
11+
version_toolchain: "nightly-2024-04-17"

‎.github/workflows/ci-risc0.yml

+6-6
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,9 @@ on:
1212
merge_group:
1313

1414
jobs:
15-
build-test-risc0:
16-
name: Build and test risc0
17-
uses: ./.github/workflows/ci-build-test-reusable.yml
18-
with:
19-
version_name: "risc0"
20-
version_toolchain: "stable"
15+
build-test-risc0:
16+
name: Build and test risc0
17+
uses: ./.github/workflows/ci-build-test-reusable.yml
18+
with:
19+
version_name: "risc0"
20+
version_toolchain: "stable"

‎.github/workflows/ci-sgx-hardware.yml

+5-5
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ jobs:
1111
TARGET: sgx
1212
CI: 1
1313
EDMM: 0
14-
14+
1515
steps:
1616
- uses: actions/checkout@v4
1717
with:
@@ -21,15 +21,15 @@ jobs:
2121
with:
2222
toolchain: stable
2323
profile: minimal
24-
24+
2525
- name: Install cargo-binstall
2626
uses: cargo-bins/[email protected]
27-
27+
2828
- name: Install sgx
2929
run: make install
30-
30+
3131
- name: Build sgx prover
3232
run: make build
33-
33+
3434
- name: Test sgx prover
3535
run: make test

‎.github/workflows/ci-sp1.yml

+6-6
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,9 @@ on:
1212
merge_group:
1313

1414
jobs:
15-
build-test-sgx:
16-
name: Build and test sp1
17-
uses: ./.github/workflows/ci-build-test-reusable.yml
18-
with:
19-
version_name: "sp1"
20-
version_toolchain: "nightly-2024-04-18"
15+
build-test-sgx:
16+
name: Build and test sp1
17+
uses: ./.github/workflows/ci-build-test-reusable.yml
18+
with:
19+
version_name: "sp1"
20+
version_toolchain: "nightly-2024-04-18"

‎.github/workflows/openapi-deploy.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ jobs:
4444
if: github.ref == 'refs/heads/main'
4545
uses: actions/upload-pages-artifact@v2
4646
with:
47-
path: './openapi'
47+
path: "./openapi"
4848

4949
- name: Deploy to GitHub Pages
5050
if: github.ref == 'refs/heads/main'

‎.gitignore

+7
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,13 @@ target/
3838
# MSVC Windows builds of rustc generate these, which store debugging information
3939
*.pdb
4040

41+
# SQLite
42+
# -----------------------------------------------------------------------------------------
43+
*.sqlite
44+
*.sqlite-shm
45+
*.sqlite-wal
46+
*.sqlite-journal
47+
4148
# Temp files, swap, debug, log, perf, cache
4249
# -----------------------------------------------------------------------------------------
4350
*.swp

‎Cargo.lock

+119-3
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml

+8-1
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ members = [
1414
"provers/sgx/setup",
1515
"pipeline",
1616
"core",
17+
"task_manager",
1718
]
1819

1920
# Always optimize; building and running the guest takes much longer without optimization.
@@ -35,6 +36,7 @@ opt-level = 3
3536
# raiko
3637
raiko-lib = { path = "./lib", features = ["std"] }
3738
raiko-core = { path = "./core" }
39+
raiko-task-manager = { path = "./task_manager" }
3840

3941
# reth
4042
reth-primitives = { git = "https://github.com/taikoxyz/taiko-reth.git", branch = "v1.0.0-rc.2-taiko", default-features = false, features = ["alloy-compat", "taiko"] }
@@ -109,6 +111,7 @@ base64-serde = "0.7.0"
109111
base64 = "0.21.7"
110112
libflate = { version = "2.0.0" }
111113
typetag = { version = "0.2.15" }
114+
num_enum = "0.7.2"
112115

113116
# tracing, logging
114117
tracing = "0.1"
@@ -135,6 +138,7 @@ tokio = { version = "^1.23", features = ["full"] }
135138
hyper = { version = "0.14.27", features = ["server"] }
136139
reqwest = { version = "0.11.22", features = ["json"] }
137140
url = "2.5.0"
141+
async-trait = "0.1.80"
138142

139143
# crypto
140144
kzg = { package = "rust-kzg-zkcrypto", git = "https://github.com/brechtpd/rust-kzg.git", branch = "sp1-patch", default-features = false }
@@ -156,6 +160,9 @@ anyhow = "1.0"
156160
thiserror = "1.0"
157161
thiserror-no-std = "2.0.2"
158162

163+
# SQLite
164+
rusqlite = { version = "0.31.0", features = ["bundled"] }
165+
159166
# misc
160167
hashbrown = { version = "0.14", features = ["inline-more"] }
161168
tempfile = "3.8"
@@ -180,4 +187,4 @@ revm = { git = "https://github.com/taikoxyz/revm.git", branch = "v36-taiko" }
180187
revm-primitives = { git = "https://github.com/taikoxyz/revm.git", branch = "v36-taiko" }
181188
revm-precompile = { git = "https://github.com/taikoxyz/revm.git", branch = "v36-taiko" }
182189
secp256k1 = { git = "https://github.com/CeciliaZ030/rust-secp256k1", branch = "sp1-patch" }
183-
blst = { git = "https://github.com/CeciliaZ030/blst.git", branch = "v0.3.12-serialize" }
190+
blst = { git = "https://github.com/CeciliaZ030/blst.git", branch = "v0.3.12-serialize" }

0 commit comments

Comments
 (0)