Skip to content

Commit b319dda

Browse files
Spicebench Streaming (#251)
* Spicebench streaming: postgres/dynamodb/mongodb * WIP * Improvements * Rename postgres * Add duckdb * Fix * Fix * Fix * Fix * Fixes * Fid * Fix * chore: auto-fix cargo fmt + clippy * Fix * Fix * Fix * Fix * Fix * Add logging * Add logging * Fix * Fix * Fix * Add mongodb and dynamodb * Fix * Add building drivers * Add adbc drivers * Add adbc drivers * Fix * Fix * Fix * Fix * chore: auto-fix cargo fmt + clippy * Fix * Properly handle sigint * Fix * Fix * chore: auto-fix cargo fmt + clippy * Improvements * Add image_tage * Proper spidapter_image_tag * Fix * Fix * Increase parallelization * Fix * Add more disk * Fix disk size * Fix * Fix dynamodb * Add dynamodb logging * Add --auto-load-complete * Fix * Add scale_factor * Add mongo to run * Fix * Fix * chore: auto-fix cargo fmt + clippy * Fix * Make mongo work * chore: auto-fix cargo fmt + clippy * Use mongo8 * chore: auto-fix cargo fmt + clippy * Lint * Fix * Fix * Fix * Remove files * Remove extra code * Fix * Fix * Fix * Fix * Fix * Fix * Fix databricks * Fix * Fix * Add debug workflow * Add ec2_disk_size_gb * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Remove debezium * Fix * Fix * Fix * Fix * Fix * New workflows * Fix * chore: auto-fix cargo fmt + clippy * Add validation timeout * Use spidapter scenarios * Rename * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Fix * chore: auto-fix cargo fmt + clippy --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
1 parent 5e8c3a5 commit b319dda

23 files changed

Lines changed: 2765 additions & 565 deletions

File tree

.github/workflows/pr.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -160,4 +160,4 @@ jobs:
160160
--scale-factor 0.001 \
161161
--etl-region us-east-1 \
162162
--system-adapter-stdio-cmd docker \
163-
--system-adapter-stdio-args "run -i -e SPICEAI_API_KEY -e SPICE_CLOUD_API_URL -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e SPIDAPTER_ICEBERG_REGION -e SPIDAPTER_ICEBERG_CATALOG_FROM ghcr.io/spiceai/spidapter:latest stdio --verbose --channel nightly"
163+
--system-adapter-stdio-args "run -i -e SPICEAI_API_KEY -e SPICE_CLOUD_API_URL -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e SPIDAPTER_ICEBERG_REGION -e SPIDAPTER_ICEBERG_CATALOG_FROM -e SPIDAPTER_CHANNEL=nightly -e SPIDAPTER_SCENARIO=cayenne ghcr.io/spiceai/spidapter:latest stdio --verbose"

.github/workflows/run_spicebench.yml

Lines changed: 67 additions & 163 deletions
Large diffs are not rendered by default.

.github/workflows/run_spicebench_debug_spice_cloud.yml

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,7 @@ jobs:
241241
ADAPTER_CMD="docker"
242242
ADAPTER_DOCKER_OPTS="run -i -e SPIDAPTER_EXECUTOR_REPLICAS=${EXECUTOR_REPLICAS} -e SPICEAI_API_KEY -e SPICE_CLOUD_API_URL -e AWS_ACCESS_KEY_ID=${S3_AWS_ACCESS_KEY_ID} -e AWS_SECRET_ACCESS_KEY=${S3_AWS_SECRET_ACCESS_KEY} -e SPIDAPTER_ICEBERG_REGION -e SPIDAPTER_ICEBERG_CATALOG_FROM -e SCHEDULER_STATE_LOCATION"
243243
244+
ADAPTER_DOCKER_OPTS="${ADAPTER_DOCKER_OPTS} -e SPIDAPTER_SCENARIO=cayenne"
244245
ADAPTER_DOCKER_OPTS="${ADAPTER_DOCKER_OPTS} -e SPIDAPTER_APP_MEMORY_LIMIT=${SPIDAPTER_APP_MEMORY_LIMIT}"
245246
ADAPTER_DOCKER_OPTS="${ADAPTER_DOCKER_OPTS} -e SPIDAPTER_EXECUTOR_MEMORY_LIMIT=${SPIDAPTER_EXECUTOR_MEMORY_LIMIT}"
246247
@@ -261,13 +262,14 @@ jobs:
261262
ADAPTER_DOCKER_OPTS="${ADAPTER_DOCKER_OPTS} -e SPIDAPTER_APP_STORAGE_SIZE_GB=3 -e SPIDAPTER_EXECUTOR_STORAGE_SIZE_GB=2 -e SPIDAPTER_CAYENNE_DATA_DIR=/data/data -e SPIDAPTER_CAYENNE_METADATA_DIR=/data/metadata"
262263
fi
263264
264-
# Use internal channel when a custom image is specified, otherwise nightly.
265+
# Channel flows into the scenario YAML via SPIDAPTER_CHANNEL env var.
265266
SPIDAPTER_CHANNEL="nightly"
266267
if [ -n "${CUSTOM_IMAGE_TAG}" ]; then
267268
SPIDAPTER_CHANNEL="internal"
268269
fi
270+
ADAPTER_DOCKER_OPTS="${ADAPTER_DOCKER_OPTS} -e SPIDAPTER_CHANNEL=${SPIDAPTER_CHANNEL}"
269271
270-
ADAPTER_ARGS="${ADAPTER_DOCKER_OPTS} ghcr.io/spiceai/spidapter:${{ github.event.inputs.spidapter_version || 'latest' }} stdio --verbose --channel ${SPIDAPTER_CHANNEL}"
272+
ADAPTER_ARGS="${ADAPTER_DOCKER_OPTS} ghcr.io/spiceai/spidapter:${{ github.event.inputs.spidapter_version || 'latest' }} stdio --verbose"
271273
ADAPTER_ENVS=""
272274
273275
NO_TEARDOWN_ARG=""

.github/workflows/run_spicebench_debug_streaming.yml

Lines changed: 119 additions & 127 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)