Skip to content

Commit ac0ce0b

Browse files
cjdsellerscjdsellers
committed
Update build tooling and CI/CD pipeline
- Upgrade to Python 3.13 and fully migrate to uv - Upgrade security hardening for GitHub actions - Add zizmor workflow auditing, ruff formatting, and docformatter - Fix deprecated ruff/mypy configs and standardize on ruff - Remove use of black
1 parent 183e98c commit ac0ce0b

File tree

11 files changed

+698
-254
lines changed

11 files changed

+698
-254
lines changed

‎.github/workflows/build.yml‎

Lines changed: 57 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2,25 +2,71 @@ name: build
22

33
# Build and test the NautilusTrader example data image
44

5+
permissions: # Principle of least privilege
6+
contents: read
7+
58
on:
69
push:
710
branches: [main]
811
pull_request:
9-
branches: [main]
12+
branches: ['*']
1013

1114
jobs:
15+
pre-commit:
16+
runs-on: ubuntu-latest
17+
steps:
18+
# https://github.com/step-security/harden-runner
19+
- uses: step-security/harden-runner@f4a75cfd619ee5ce8d5b864b0d183aff3c69b55a # v2.13.1
20+
with:
21+
egress-policy: audit
22+
23+
- name: Checkout repository
24+
# https://github.com/actions/checkout
25+
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
26+
with:
27+
persist-credentials: false
28+
29+
- name: Set up Python
30+
# https://github.com/actions/setup-python
31+
uses: actions/setup-python@cae9d6a5c961a2c267bc955ecf8d27bc955b2e83 # v5.4.0
32+
with:
33+
python-version: "3.13"
34+
35+
- name: Install uv
36+
# https://github.com/astral-sh/setup-uv
37+
uses: astral-sh/setup-uv@1b6072f24a1f5e1bf6797b14bb96e52cf5e1f027 # v5.0.1
38+
39+
- name: Install dependencies
40+
run: uv sync --all-groups
41+
42+
- name: Run pre-commit
43+
run: uv run pre-commit run --all-files
44+
1245
docker-image:
1346
runs-on: ubuntu-latest
47+
needs:
48+
- pre-commit
49+
permissions:
50+
contents: read
51+
packages: write
1452
steps:
15-
- name: Set up QEMU
16-
# https://github.com/docker/setup-qemu-action
17-
uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392 # v3.6.0
53+
# https://github.com/step-security/harden-runner
54+
- uses: step-security/harden-runner@f4a75cfd619ee5ce8d5b864b0d183aff3c69b55a # v2.13.1
55+
with:
56+
egress-policy: audit
57+
58+
- name: Checkout repository
59+
# https://github.com/actions/checkout
60+
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
61+
with:
62+
persist-credentials: false
1863

1964
- name: Set up Docker Buildx
2065
# https://github.com/docker/setup-buildx-action
2166
uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0
2267

2368
- name: Login to GHCR
69+
if: github.event_name == 'push'
2470
# https://github.com/docker/login-action
2571
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
2672
with:
@@ -33,19 +79,19 @@ jobs:
3379
run: |
3480
echo "current_branch=${GITHUB_REF#refs/heads/}" >> $GITHUB_OUTPUT
3581
36-
- name: image name
37-
run: echo ghcr.io/${{ github.repository }}:${{ steps.branch-name.outputs.current_branch }}
38-
39-
- name: Build backend
82+
- name: Build and push Docker image
4083
id: docker_build
4184
# https://github.com/docker/build-push-action
4285
uses: docker/build-push-action@471d1dc4e07e5cdedd4c2171150001c434f0b7a4 # v6.15.0
4386
with:
44-
file: "Dockerfile"
45-
push: true
87+
context: .
88+
file: Dockerfile
89+
platforms: linux/amd64
90+
push: ${{ github.event_name == 'push' }}
4691
tags: ghcr.io/${{ github.repository }}:${{ steps.branch-name.outputs.current_branch }}
4792
cache-from: type=gha
48-
cache-to: type=gha
93+
cache-to: type=gha,mode=max
4994

5095
- name: Image digest
96+
if: github.event_name == 'push'
5197
run: echo ${{ steps.docker_build.outputs.digest }}

‎.pre-commit-config.yaml‎

Lines changed: 40 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -3,16 +3,16 @@ repos:
33
# General checks
44
##############################################################################
55
- repo: https://github.com/pre-commit/pre-commit-hooks
6-
rev: v5.0.0
6+
rev: v6.0.0
77
hooks:
8-
- id: fix-encoding-pragma
9-
args: [--remove]
108
- id: end-of-file-fixer
11-
types_or: [python]
9+
types_or: [python, markdown]
10+
- id: mixed-line-ending
1211
- id: trailing-whitespace
13-
types_or: [python]
12+
types_or: [python, markdown]
1413
- id: debug-statements
1514
- id: detect-private-key
15+
- id: check-added-large-files
1616
- id: check-builtin-literals
1717
- id: check-case-conflict
1818
- id: check-executables-have-shebangs
@@ -29,37 +29,60 @@ repos:
2929
- id: codespell
3030
description: Checks for common misspellings.
3131
types_or: [python, cython, rst, markdown]
32+
args: ["-L", "ACN,crate,ot,socio-economic,zar"]
33+
34+
##############################################################################
35+
# Custom hooks
36+
##############################################################################
37+
- repo: local
38+
hooks:
39+
- id: zizmor
40+
name: zizmor (audit GitHub actions)
41+
description: Audit GitHub Actions workflows
42+
language: python
43+
additional_dependencies: [zizmor]
44+
entry: zizmor
45+
files: '^\.github/workflows/.*\.ya?ml$'
46+
pass_filenames: false
47+
args: ["--no-online-audits", "--min-severity", "medium", ".github/workflows"]
3248

3349
##############################################################################
3450
# Python formatting and linting
3551
##############################################################################
3652
- repo: https://github.com/asottile/add-trailing-comma
37-
rev: v3.1.0
53+
rev: v3.2.0
3854
hooks:
3955
- id: add-trailing-comma
4056
name: add-trailing-comma
4157
types: [python]
4258

43-
- repo: https://github.com/psf/black
44-
rev: 25.1.0
45-
hooks:
46-
- id: black
47-
types_or: [python, pyi]
48-
entry: "black"
49-
5059
- repo: https://github.com/astral-sh/ruff-pre-commit
51-
rev: v0.11.4
60+
rev: v0.13.1
5261
hooks:
5362
- id: ruff
5463
args: ["--fix"]
64+
- id: ruff-format
65+
66+
- repo: https://github.com/PyCQA/docformatter
67+
rev: v1.7.7
68+
hooks:
69+
- id: docformatter
70+
additional_dependencies: [tomli]
71+
args: [
72+
"--black",
73+
"--make-summary-multi-line",
74+
"--pre-summary-newline",
75+
"--blank",
76+
"--recursive",
77+
"--in-place",
78+
]
5579

5680
- repo: https://github.com/pre-commit/mirrors-mypy
57-
rev: v1.15.0
81+
rev: v1.18.2
5882
hooks:
5983
- id: mypy
60-
args: [--no-strict-optional, --ignore-missing-imports, --warn-no-return, --explicit-package-bases]
84+
args: ["--config", "pyproject.toml"]
6185
additional_dependencies: [types-pytz, types-redis, types-toml, types-requests, msgspec]
62-
exclude: "nautilus_server/backend/api_v1/endpoints/models*"
6386

6487
- repo: https://github.com/kynan/nbstripout
6588
rev: 0.8.1

‎Dockerfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM python:3.12-slim AS base
1+
FROM python:3.13-slim AS base
22
ENV PYTHONUNBUFFERED=1 \
33
PYTHONDONTWRITEBYTECODE=1 \
44
PIP_NO_CACHE_DIR=off \
@@ -29,7 +29,7 @@ FROM base AS application
2929
ENV CATALOG_PATH=/catalog
3030

3131
# Copy python environment from builder
32-
COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
32+
COPY --from=builder /usr/local/lib/python3.13/site-packages /usr/local/lib/python3.13/site-packages
3333
COPY ./nautilus_data $PYSETUP_PATH/nautilus_data
3434

3535
# Ensure the catalog and backtest directories exist

‎Makefile‎

Lines changed: 114 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,125 @@
1+
# Variables
2+
# -----------------------------------------------------------------------------
13
PROJECT?=nautechsystems/nautilus_data
24
REGISTRY?=ghcr.io/
3-
IMAGE?=${REGISTRY}${PROJECT}
5+
IMAGE?=$(REGISTRY)$(PROJECT)
46
GIT_TAG:=$(shell git rev-parse --abbrev-ref HEAD)
5-
IMAGE_FULL?=${IMAGE}:${GIT_TAG}
6-
PYTHON_EXECUTABLE:=$(shell which python3)
7-
NAUTILUS_PATH:="${HOME}/.nautilus"
7+
IMAGE_FULL?=$(IMAGE):$(GIT_TAG)
88

9-
########################################
10-
# Docker development commands
11-
########################################
12-
docker-build:
13-
(docker pull ${IMAGE} || true)
14-
(docker build --platform linux/x86_64 -t ${IMAGE_FULL} .)
9+
V = 0 # 0 / 1 - verbose mode
10+
Q = $(if $(filter 1,$V),,@) # Quiet mode, suppress command output
11+
M = $(shell printf "$(BLUE)>$(RESET)") # Message prefix for commands
1512

16-
docker-build-force:
17-
(docker build --no-cache -t ${IMAGE_FULL} ./.. )
13+
# > Colors
14+
RED := $(shell tput -Txterm setaf 1)
15+
GREEN := $(shell tput -Txterm setaf 2)
16+
YELLOW := $(shell tput -Txterm setaf 3)
17+
BLUE := $(shell tput -Txterm setaf 4)
18+
PURPLE := $(shell tput -Txterm setaf 5)
19+
CYAN := $(shell tput -Txterm setaf 6)
20+
GRAY := $(shell tput -Txterm setaf 7)
21+
RESET := $(shell tput -Txterm sgr0)
1822

19-
docker-push:
20-
(docker push ${IMAGE_FULL} )
23+
.DEFAULT_GOAL := help
2124

22-
########################################
23-
# Development commands
24-
########################################
25+
#== Installation
2526

26-
update:
27-
poetry update
27+
.PHONY: install
28+
install: #-- Install dependencies using uv
29+
$(info $(M) Installing dependencies...)
30+
$Q uv sync
2831

29-
pre-commit:
30-
pre-commit run --all-files
32+
.PHONY: install-dev
33+
install-dev: #-- Install all development dependencies
34+
$(info $(M) Installing development dependencies...)
35+
$Q uv sync --all-groups
3136

32-
clean:
37+
.PHONY: update
38+
update: #-- Update dependencies using uv
39+
$(info $(M) Updating dependencies...)
40+
$Q uv sync --upgrade
41+
42+
#== Clean
43+
44+
.PHONY: clean
45+
clean: clean-caches #-- Clean all caches and build artifacts
46+
$(info $(M) Cleaning all caches and build artifacts...)
47+
$Q rm -rf dist build 2>/dev/null || true
48+
49+
.PHONY: clean-caches
50+
clean-caches: #-- Clean pytest, mypy, ruff, and uv caches
51+
$Q rm -rf .pytest_cache .mypy_cache .ruff_cache 2>/dev/null || true
52+
$Q find . -type d -name "__pycache__" -not -path "./.venv*" -print0 | xargs -0 -r rm -rf
53+
$Q find . -type f -name "*.pyc" -not -path "./.venv*" -print0 | xargs -0 -r rm -f
54+
55+
.PHONY: distclean
56+
distclean: clean #-- Nuclear clean - remove all untracked files (requires FORCE=1)
57+
@[ "$$FORCE" = 1 ] || { echo "Pass FORCE=1 to really nuke"; exit 1; }
58+
@echo "⚠️ nuking working tree (git clean -fxd)…"
3359
git clean -fxd
60+
61+
#== Code Quality
62+
63+
.PHONY: format
64+
format: #-- Format code using ruff
65+
$(info $(M) Formatting code with ruff...)
66+
$Q uv run --no-sync ruff format .
67+
68+
.PHONY: pre-commit
69+
pre-commit: #-- Run all pre-commit hooks on all files
70+
$(info $(M) Running pre-commit hooks...)
71+
$Q uv run --no-sync pre-commit run --all-files
72+
73+
.PHONY: ruff
74+
ruff: #-- Run ruff linter with automatic fixes
75+
$(info $(M) Running ruff linter with fixes...)
76+
$Q uv run --no-sync ruff check . --fix
77+
78+
.PHONY: mypy
79+
mypy: #-- Run mypy static type checker
80+
$(info $(M) Running mypy type checker...)
81+
$Q uv run --no-sync mypy .
82+
83+
#== Testing
84+
85+
.PHONY: pytest
86+
pytest: #-- Run Python tests with pytest
87+
$(info $(M) Running pytest...)
88+
$Q uv run --no-sync pytest
89+
90+
#== Docker
91+
92+
.PHONY: docker-build
93+
docker-build: #-- Build Docker image
94+
$(info $(M) Building Docker image $(IMAGE_FULL)...)
95+
$Q docker pull $(IMAGE) || true
96+
$Q docker build --platform linux/x86_64 -t $(IMAGE_FULL) .
97+
98+
.PHONY: docker-build-force
99+
docker-build-force: #-- Force rebuild Docker image without cache
100+
$(info $(M) Force rebuilding Docker image $(IMAGE_FULL)...)
101+
$Q docker build --no-cache -t $(IMAGE_FULL) .
102+
103+
.PHONY: docker-push
104+
docker-push: #-- Push Docker image to registry
105+
$(info $(M) Pushing Docker image $(IMAGE_FULL)...)
106+
$Q docker push $(IMAGE_FULL)
107+
108+
#== Help
109+
110+
.PHONY: help
111+
help: #-- Show this help message
112+
@echo "$(CYAN)Nautilus Data Makefile$(RESET)"
113+
@echo ""
114+
@echo "$(YELLOW)Usage:$(RESET)"
115+
@echo " make [target] [V=1] [OPTION=value]"
116+
@echo ""
117+
@echo "$(YELLOW)Available targets:$(RESET)"
118+
@grep -E '^[a-zA-Z0-9_%/-]+:.*#--' $(MAKEFILE_LIST) | \
119+
awk 'BEGIN {FS = ":.*#--"}; \
120+
{gsub(/^[^:]*:/, "", $$1); \
121+
printf " $(GREEN)%-20s$(RESET) %s\n", $$1, $$2}'
122+
@echo ""
123+
@echo "$(YELLOW)Options:$(RESET)"
124+
@echo " $(BLUE)V=1$(RESET) Enable verbose output"
125+
@echo " $(BLUE)FORCE=1$(RESET) Required for distclean target"

‎bench_data/check_invariant.py‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,8 @@
55

66
def check_file(file_name):
77
"""
8-
Check if the 'start_ts' column is in ascending order and 'end_ts' for a row
9-
groups comes before 'start_ts' of the next row group.
10-
8+
Check if the 'start_ts' column is in ascending order and 'end_ts' for a row groups
9+
comes before 'start_ts' of the next row group.
1110
"""
1211
df = pd.read_csv(file_name)
1312

@@ -18,7 +17,7 @@ def check_file(file_name):
1817
# Check if 'end_ts' for a row comes before 'start_ts' of the next row
1918
for i in range(1, len(df)):
2019
if df.loc[i - 1, "end_ts"] > df.loc[i, "start_ts"]:
21-
print(f"Row {i-1} and {i} fail the check:")
20+
print(f"Row {i - 1} and {i} fail the check:")
2221
print(df.loc[[i - 1, i]])
2322

2423

‎bench_data/extract_groups.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,9 @@
4646

4747

4848
def write_parquet_with_row_group(input_file, output_file, rows_per_row_group):
49-
"""Write a Parquet file with specified row group size."""
49+
"""
50+
Write a Parquet file with specified row group size.
51+
"""
5052
df = pd.read_parquet(input_file)
5153

5254
schema = quote_tick_schema if "quotes" in input_file else trade_tick_schema

0 commit comments

Comments
 (0)