Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 68 additions & 6 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,46 @@ jobs:
fail-fast: false
matrix:
include:
# flows-collab smoke test (linux)
- scenario: flows-collab
platform: linux-x86_64
runner: namespace-profile-linux-medium
client_mode: rust
tauri_binary: src-tauri/target/release/bv-desktop
run_prefix: "xvfb-run -a"

# flows-collab smoke test (windows)
- scenario: flows-collab
platform: windows-x86_64
runner: namespace-profile-windows-medium
client_mode: rust
tauri_binary: src-tauri/target/release/bv-desktop
run_prefix: ""

# jupyter-collab smoke test (linux)
- scenario: jupyter-collab
platform: linux-x86_64
runner: namespace-profile-linux-large
client_mode: rust
tauri_binary: src-tauri/target/release/bv-desktop
run_prefix: "xvfb-run -a"

# jupyter-collab smoke test (macOS)
- scenario: jupyter-collab
platform: macos-arm64
runner: namespace-profile-mac-medium
client_mode: rust
tauri_binary: src-tauri/target/release/bv-desktop
run_prefix: ""

# jupyter-collab smoke test (windows)
- scenario: jupyter-collab
platform: windows-x86_64
runner: namespace-profile-windows-medium
client_mode: rust
tauri_binary: src-tauri/target/release/bv-desktop
run_prefix: ""

# pipelines-multiparty-flow smoke test (linux)
- scenario: pipelines-multiparty-flow
platform: linux-x86_64
Expand Down Expand Up @@ -105,7 +145,7 @@ jobs:
path: |
${{ github.workspace }}/../biovault
${{ github.workspace }}/../syftbox-sdk
${{ github.workspace }}/../syft-crypto-core
${{ github.workspace }}/../syftbox-crypto
${{ github.workspace }}/../syftbox
${{ github.workspace }}/../biovault-beaver
${{ github.workspace }}/../sbenv
Expand All @@ -116,12 +156,20 @@ jobs:
workspace-deps-${{ runner.os }}-

- name: Setup workspace
if: runner.os != 'Windows'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BV_SKIP_SYQURE: ${{ contains(matrix.scenario, 'syqure') && '0' || '1' }}
run: |
chmod +x scripts/setup-workspace.sh
./scripts/setup-workspace.sh
- name: Setup workspace (Windows)
if: runner.os == 'Windows'
shell: pwsh
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BV_SKIP_SYQURE: ${{ contains(matrix.scenario, 'syqure') && '0' || '1' }}
run: .\win.ps1 ./scripts/setup-workspace.sh

- name: Setup Rust
uses: dtolnay/rust-toolchain@stable
Expand Down Expand Up @@ -442,7 +490,7 @@ jobs:
run: ${{ matrix.run_prefix }} ./test-scenario.sh --flows-pause-resume

- name: Run flows collab scenario
if: matrix.scenario == 'flows-collab'
if: runner.os != 'Windows' && matrix.scenario == 'flows-collab'
env:
SKIP_PLAYWRIGHT_INSTALL: "1"
AUTO_REBUILD_TAURI: "0"
Expand All @@ -452,9 +500,16 @@ jobs:
else
./test-scenario.sh --flows-collab
fi
- name: Run flows collab scenario (Windows)
if: runner.os == 'Windows' && matrix.scenario == 'flows-collab'
shell: pwsh
env:
SKIP_PLAYWRIGHT_INSTALL: "1"
AUTO_REBUILD_TAURI: "0"
run: .\win.ps1 ./test-scenario.sh --flows-collab

- name: Run jupyter collab scenario
if: matrix.scenario == 'jupyter-collab'
if: runner.os != 'Windows' && matrix.scenario == 'jupyter-collab'
env:
SKIP_PLAYWRIGHT_INSTALL: "1"
AUTO_REBUILD_TAURI: "0"
Expand All @@ -464,6 +519,13 @@ jobs:
else
./test-scenario.sh --jupyter-collab biovault-beaver/notebooks/02-advanced-features.json
fi
- name: Run jupyter collab scenario (Windows)
if: runner.os == 'Windows' && matrix.scenario == 'jupyter-collab'
shell: pwsh
env:
SKIP_PLAYWRIGHT_INSTALL: "1"
AUTO_REBUILD_TAURI: "0"
run: .\win.ps1 ./test-scenario.sh --jupyter-collab biovault-beaver/notebooks/02-advanced-features.json

- name: Run pipelines multiparty flow scenario
if: matrix.scenario == 'pipelines-multiparty-flow'
Expand Down Expand Up @@ -531,7 +593,7 @@ jobs:
path: |
${{ github.workspace }}/../biovault
${{ github.workspace }}/../syftbox-sdk
${{ github.workspace }}/../syft-crypto-core
${{ github.workspace }}/../syftbox-crypto
${{ github.workspace }}/../syftbox
${{ github.workspace }}/../biovault-beaver
${{ github.workspace }}/../sbenv
Expand Down Expand Up @@ -705,7 +767,7 @@ jobs:
path: |
${{ github.workspace }}/../biovault
${{ github.workspace }}/../syftbox-sdk
${{ github.workspace }}/../syft-crypto-core
${{ github.workspace }}/../syftbox-crypto
${{ github.workspace }}/../syftbox
${{ github.workspace }}/../biovault-beaver
${{ github.workspace }}/../sbenv
Expand Down Expand Up @@ -853,7 +915,7 @@ jobs:
path: |
${{ github.workspace }}/../biovault
${{ github.workspace }}/../syftbox-sdk
${{ github.workspace }}/../syft-crypto-core
${{ github.workspace }}/../syftbox-crypto
${{ github.workspace }}/../syftbox
${{ github.workspace }}/../biovault-beaver
${{ github.workspace }}/../sbenv
Expand Down
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ templates-dev/
# Sibling repo symlinks (created by setup-workspace.sh)
/biovault/
/syftbox-sdk/
/syft-crypto-core/
/syftbox-crypto/
/syftbox/
/biovault-beaver/
/sbenv/
Expand All @@ -64,7 +64,7 @@ bioscript/
biovault-beaver/
syftbox/
syftbox-sdk/
syft-crypto-core/
syftbox-crypto/
sbenv/

# repo tool workspace metadata
Expand Down
2 changes: 1 addition & 1 deletion .prettierignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ biovault
bioscript
biovault-beaver
sbenv
syft-crypto-core
syftbox-crypto
syftbox
syftbox-sdk
syqure
Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ This repo is a kind of mono-repo for BioVault but uses Google's repo tool so the

- Preferred layout is sibling repos at the repo root:
`biovault/`, `biovault-beaver/`, `bioscript/`, `syftbox/`, `syftbox-sdk/`,
`syft-crypto-core/`, `sbenv/`
`syftbox-crypto/`, `sbenv/`
- Scripts and Rust code accept overrides via env vars (examples):
`WORKSPACE_ROOT`, `BIOVAULT_DIR`, `BIOVAULT_BEAVER_DIR`, `SYFTBOX_DIR`,
`SYFTBOX_SDK_DIR`, `SBENV_DIR`
Expand All @@ -26,7 +26,7 @@ This repo is a kind of mono-repo for BioVault but uses Google's repo tool so the
### Nested repo (libsignal)

- The libsignal repo is checked out under
`syft-crypto-core/vendor/libsignal-protocol-syft`
`syftbox-crypto/vendor/libsignal-protocol-syft`
- It is pinned in `manifest.xml` to avoid submodule workflows.

### Notes for CI
Expand All @@ -39,7 +39,7 @@ This repo is a kind of mono-repo for BioVault but uses Google's repo tool so the
### Dependency Graph

```
syft-crypto-core (foundational crypto)
syftbox-crypto (foundational crypto)
syftbox-sdk (depends on syft-crypto-protocol)
Expand All @@ -54,7 +54,7 @@ biovault-beaver (independent Python, but uses syftbox conventions)

When making changes that span multiple repos, release in this order:

1. **syft-crypto-core** - Foundational crypto library
1. **syftbox-crypto** - Foundational crypto library

- Contains `syft-crypto-protocol` crate
- Must be released first if crypto changes
Expand Down Expand Up @@ -103,7 +103,7 @@ cd .. && git add -A && git commit -m "feat: integrate X"

When releasing, update versions in this order:

1. `syft-crypto-core/protocol/Cargo.toml` - bump version
1. `syftbox-crypto/protocol/Cargo.toml` - bump version
2. `syftbox-sdk/Cargo.toml` - bump version, update `syft-crypto-protocol` dep
3. `biovault/cli/Cargo.toml` - bump version, update `syftbox-sdk` dep
4. `src-tauri/Cargo.toml` - bump version (auto-triggers release)
Expand Down Expand Up @@ -137,7 +137,7 @@ Each repo has a `lint.sh` script that runs all checks in parallel:
| biovault-desktop | Rust + JS/TS | `cargo fmt` + `clippy` + `prettier` + `eslint` |
| biovault | Rust | `cargo fmt` + `clippy` |
| syftbox-sdk | Rust | `cargo fmt` + `clippy` |
| syft-crypto-core | Rust | `cargo fmt` + `clippy` |
| syftbox-crypto | Rust | `cargo fmt` + `clippy` |
| biovault-beaver | Python | `ruff format` + `ruff check` + `mypy` + `vulture` |

With `--test`: adds `cargo test` (Rust) or `pytest` (Python)
Expand Down
2 changes: 1 addition & 1 deletion dev-desktop-live.sh
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ export SYFTBOX_AUTH_ENABLED
export SYFTBOX_EMAIL
export SYFTBOX_CONFIG_PATH="$BIOVAULT_HOME/syftbox/config.json"
export SYFTBOX_DATA_DIR="$BIOVAULT_HOME"
export SYC_VAULT="$SYFTBOX_DATA_DIR/.syc"
export SBC_VAULT="$SYFTBOX_DATA_DIR/.sbc"

export BIOVAULT_DEV_MODE=1
export BIOVAULT_DEV_SYFTBOX=1
Expand Down
6 changes: 3 additions & 3 deletions dev-three.sh
Original file line number Diff line number Diff line change
Expand Up @@ -416,7 +416,7 @@ launch_desktop_instance() {
export SYFTBOX_SERVER_URL="$server"
export SYFTBOX_CONFIG_PATH="$config"
export SYFTBOX_DATA_DIR="$data_dir"
export SYC_VAULT="$SYFTBOX_DATA_DIR/.syc"
export SBC_VAULT="$SYFTBOX_DATA_DIR/.sbc"
export DEV_WS_BRIDGE=1
export DEV_WS_BRIDGE_PORT="$ws_port"

Expand Down Expand Up @@ -453,7 +453,7 @@ launch_desktop_instance() {
echo -e "${YELLOW} BIOVAULT_HOME: $BIOVAULT_HOME${NC}"
echo -e "${YELLOW} SYFTBOX_DATA_DIR: $SYFTBOX_DATA_DIR${NC}"
echo -e "${YELLOW} SYFTBOX_CONFIG: $SYFTBOX_CONFIG_PATH${NC}"
echo -e "${YELLOW} SYC_VAULT: $SYC_VAULT${NC}"
echo -e "${YELLOW} SBC_VAULT: $SBC_VAULT${NC}"
echo -e "${YELLOW} SyftBox backend: $BV_SYFTBOX_BACKEND${NC}"
echo -e "${YELLOW} Server: $SYFTBOX_SERVER_URL${NC}"
echo -e "${YELLOW} WS Bridge Port: $DEV_WS_BRIDGE_PORT${NC}"
Expand Down Expand Up @@ -540,7 +540,7 @@ preseed_onboarding_for_bootstrap() {
SYFTBOX_SERVER_URL="$server" \
SYFTBOX_CONFIG_PATH="$config" \
SYFTBOX_DATA_DIR="$data_dir" \
SYC_VAULT="$data_dir/.syc" \
SBC_VAULT="$data_dir/.sbc" \
"$BV_CLI_BIN" init --quiet "$email" >/dev/null
done

Expand Down
4 changes: 2 additions & 2 deletions dev-two-live.sh
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ import_bundle() {
SYFTBOX_EMAIL="$dst_email" \
SYFTBOX_SERVER_URL="$SYFTBOX_URL" \
SYFTBOX_AUTH_ENABLED="$SYFTBOX_AUTH_ENABLED" \
"$BV_CLI_BIN" syc import "$src_bundle" --expected-identity "$src_email" || true
"$BV_CLI_BIN" sbc import "$src_bundle" --expected-identity "$src_email" || true
}

ensure_bundle_under_datasites() {
Expand Down Expand Up @@ -286,7 +286,7 @@ launch_instance() {
"SYFTBOX_AUTH_ENABLED=$SYFTBOX_AUTH_ENABLED"
"SYFTBOX_CONFIG_PATH=$env_config"
"SYFTBOX_DATA_DIR=$env_data"
"SYC_VAULT=$SYFTBOX_DATA_DIR/.syc"
"SBC_VAULT=$SYFTBOX_DATA_DIR/.sbc"
"BIOVAULT_DEBUG_BANNER=1"
)
if [[ "$backend" == "process" ]]; then
Expand Down
8 changes: 4 additions & 4 deletions dev-two.sh
Original file line number Diff line number Diff line change
Expand Up @@ -398,7 +398,7 @@ launch_desktop_instance() {
export SYFTBOX_SERVER_URL="$server"
export SYFTBOX_CONFIG_PATH="$config"
export SYFTBOX_DATA_DIR="$data_dir"
export SYC_VAULT="$SYFTBOX_DATA_DIR/.syc"
export SBC_VAULT="$SYFTBOX_DATA_DIR/.sbc"
export DEV_WS_BRIDGE=1
export DEV_WS_BRIDGE_PORT="$ws_port"

Expand All @@ -411,7 +411,7 @@ launch_desktop_instance() {
echo -e "${YELLOW} BIOVAULT_HOME: $BIOVAULT_HOME${NC}"
echo -e "${YELLOW} SYFTBOX_DATA_DIR: $SYFTBOX_DATA_DIR${NC}"
echo -e "${YELLOW} SYFTBOX_CONFIG: $SYFTBOX_CONFIG_PATH${NC}"
echo -e "${YELLOW} SYC_VAULT: $SYC_VAULT${NC}"
echo -e "${YELLOW} SBC_VAULT: $SBC_VAULT${NC}"
echo -e "${YELLOW} SyftBox backend: $BV_SYFTBOX_BACKEND${NC}"
echo -e "${YELLOW} Server: $SYFTBOX_SERVER_URL${NC}"
echo -e "${YELLOW} WS Bridge Port: $DEV_WS_BRIDGE_PORT${NC}"
Expand Down Expand Up @@ -514,7 +514,7 @@ ensure_client_identity() {
syftbox_data="$(parse_data_dir "$config")"
[[ -z "$syftbox_data" ]] && { log_error "Could not read data_dir from $config"; exit 1; }

vault_dir="$home/.syc"
vault_dir="$home/.sbc"
mkdir -p "$vault_dir/keys" "$vault_dir/bundles" "$vault_dir/config"
local has_keys=0
if [[ -d "$vault_dir/keys" ]]; then
Expand Down Expand Up @@ -573,7 +573,7 @@ import_bundle_pair() {
BIOVAULT_HOME="$(client_field "$dst_email" home)" \
SYFTBOX_CONFIG_PATH="$(client_field "$dst_email" config)" \
SYFTBOX_DATA_DIR="$(parse_data_dir "$(client_field "$dst_email" config)")" \
"$BV_CLI_BIN" syc import "$bundle" --expected-identity "$src_email" \
"$BV_CLI_BIN" sbc import "$bundle" --expected-identity "$src_email" \
|| log_warn "Bundle import $src_email -> $dst_email failed (continuing)"
}

Expand Down
4 changes: 2 additions & 2 deletions docs/agent-api.json
Original file line number Diff line number Diff line change
Expand Up @@ -2280,7 +2280,7 @@
},
"readOnly": false,
"async": false,
"sideEffects": ["Stops daemons", "Deletes BIOVAULT_HOME (preserves .syc)"],
"sideEffects": ["Stops daemons", "Deletes BIOVAULT_HOME (preserves .sbc)"],
"dangerous": true
},
"reset_everything": {
Expand All @@ -2292,7 +2292,7 @@
},
"readOnly": false,
"async": false,
"sideEffects": ["Deletes all data including .syc", "Recreates empty database"],
"sideEffects": ["Deletes all data including .sbc", "Recreates empty database"],
"dangerous": true
},
"get_files": {
Expand Down
2 changes: 1 addition & 1 deletion manifest.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<project name="OpenMined/biovault" path="biovault" revision="main" />
<project name="OpenMined/biovault-beaver" path="biovault-beaver" revision="main" />
<project name="OpenMined/sbenv" path="sbenv" revision="main" />
<project name="OpenMined/syft-crypto-core" path="syft-crypto-core" revision="main" />
<project name="OpenMined/syftbox-crypto" path="syftbox-crypto" revision="main" />
<project name="OpenMined/syftbox" path="syftbox" revision="madhava/biovault" />
<project name="OpenMined/syftbox-sdk" path="syftbox-sdk" revision="main" />
<project name="madhavajay/syqure" path="syqure" revision="main" />
Expand Down
2 changes: 1 addition & 1 deletion plans/fast-webrtc.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ rg -n "burst-c1->c2\\s+PASS|Scenario completed successfully|Syqure step duration
- `Failed to read response file ... Failed to parse response JSON ...`
- Cause:
- RPC ACK scanning attempted to parse every `*.response` under scanned datasites.
- After the SYC envelope / multi-recipient rollout, many of these files are encrypted for a different identity and are expected to be unreadable by this instance.
- After the SBC envelope / multi-recipient rollout, many of these files are encrypted for a different identity and are expected to be unreadable by this instance.
- Fix implemented:
- `syftbox-sdk/src/syftbox/endpoint.rs`:
- added `check_responses_for_ids(&HashSet<String>)` to pre-filter by request-id filename before parse/decrypt.
Expand Down
Loading
Loading