Skip to content

Commit 31805f3

Browse files
wp99cpwp99cp
authored
Version 0.56.0
- chore: big refactor of the repo structure: - We switched from yarn to pnpm - We are now using a package-based structure instead of manual code linking - docs: complete restructure of docs: - better postgres schema docs - add action examples (partially completes #1759) - restructure docs (partially completes #1977) - fix outdated info (#1964) - chore: significantly reduce the image size for production images (#1965) - chore: add agents' instructions (as an alternative to #1814) - fix: improvements for new datatypes (#1758) - feat: #1936 - feat: #1935 - fix: #1923 - fix: #1733 - fix: #1088 - fix: #1965 - fix: #1162 - fix: #1926 - fix: #1963 - fix: #557 - fix: #1941 - fix: fix remaining issues of #842 - chore: #1932 - docs: #1759 - chore: dependency updates
2 parents cedbce0 + 12df380 commit 31805f3

834 files changed

Lines changed: 74070 additions & 39373 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.dockerignore‎

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,24 @@
1-
**/dist
2-
**/node_modules
3-
**/build
4-
**/.cache
1+
# Local dev files
2+
node_modules
3+
dist
4+
.env*
5+
.git
6+
.gitignore
7+
.vscode
8+
.idea
9+
10+
# Tests
11+
backend/tests
12+
!backend/tests/actions/file-hash
13+
packages/*/tests
14+
**/*.test.ts
15+
**/*.spec.ts
16+
junit.xml
17+
coverage
18+
19+
# Backend specific
20+
backend/tsconfig.tsbuildinfo
21+
backend/npm-debug.log*
22+
backend/yarn-debug.log*
23+
backend/yarn-error.log*
24+
backend/pnpm-debug.log*

‎.env‎

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
SERVER_PORT=3000
2-
QUASAR_ENDPOINT=http://localhost:3000
32

43
FRONTEND_URL=http://localhost:8003
54
DOCS_URL=http://localhost:4000
5+
BACKEND_URL=http://localhost:3000
66

77
MINIO_USER=minioadmin
88
MINIO_PASSWORD=minioadmin
@@ -13,6 +13,7 @@ MINIO_DB_BUCKET_NAME=dbdumps
1313
MINIO_ARTIFACTS_BUCKET_NAME=artifacts
1414
MINIO_ENDPOINT=localhost
1515

16+
1617
DB_HOST=database
1718
DEV=true
1819
DB_DATABASE=dbname
@@ -26,18 +27,20 @@ ENTITIES=dist/**/*.entity.js
2627
SEED=true
2728

2829
VITE_USE_FAKE_OAUTH_FOR_DEVELOPMENT=true
29-
GOOGLE_KEY_FILE=grandtourdatasets-5295745f7fab.json
30-
GOOGLE_CLIENT_ID=237481987385-av4suqf7p0qjkmvjo8bc6jdf3g6nb0u7.apps.googleusercontent.com
31-
GOOGLE_CLIENT_SECRET=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
30+
GOOGLE_KEY_FILE=google-service-account.json
3231
JWT_SECRET=SECRET
3332

34-
# this should be replaced with your own oauth credentials
35-
GITHUB_CLIENT_ID=some-client-id
36-
GITHUB_CLIENT_SECRET=some-client-secret
33+
GOOGLE_CLIENT_ID=
34+
GOOGLE_CLIENT_SECRET=
35+
36+
GITHUB_CLIENT_ID=
37+
GITHUB_CLIENT_SECRET=
3738

38-
# can be left empty if you don't want to use docker hub
3939
DOCKER_HUB_USERNAME=
4040
DOCKER_HUB_PASSWORD=
4141
VITE_DOCKER_HUB_NAMESPACE=
4242

43-
ARTIFACTS_UPLOADER_IMAGE=rslethz/grandtour-datasets:artifact-uploader-latest
43+
ARTIFACTS_UPLOADER_IMAGE=rslethz/kleinkram:artifact-uploader-latest
44+
45+
# Docker socket group ID (run: stat -c '%g' /var/run/docker.sock)
46+
DOCKER_GID=984

‎.github/AGENTS.md‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
# AI Agent Instructions
2+
3+
This document provides a quick reference for AI agents working on the Kleinkram project. For detailed information, always refer to the official documentation in the `../docs/` directory.
4+
5+
## 🚀 Quick Start (Launch)
6+
7+
To launch the full application stack:
8+
9+
```bash
10+
docker compose up --build --watch -d
11+
```
12+
13+
**Access Points:**
14+
15+
- **API Server**: [http://localhost:3000](http://localhost:3000)
16+
- **Frontend**: [http://localhost:8003](http://localhost:8003)
17+
- **MinIO Console**: [http://localhost:9001](http://localhost:9001)
18+
- **Documentation**: [http://localhost:4000](http://localhost:4000)
19+
20+
> [!NOTE]
21+
> See [../docs/development/getting-started.md](../docs/development/getting-started.md) for detailed setup instructions.
22+
23+
## 🛠️ Development Workflow
24+
25+
### API Interaction
26+
27+
**IMPORTANT**: All API calls to the backend must include the `kleinkram-client-version` header matching the current app version (e.g., `0.56.0`).
28+
29+
```typescript
30+
headers: {
31+
'kleinkram-client-version': '0.56.0', // Replace with actual version from package.json
32+
...
33+
}
34+
```
35+
36+
**Exception:** The `/api/health` endpoint does not require this header.
37+
38+
### Formatting & Linting
39+
40+
Run these commands before submitting changes:
41+
42+
```bash
43+
# Format Python code
44+
black .
45+
46+
# Format TypeScript/JavaScript/JSON/Markdown
47+
yarn run prettier
48+
49+
# Run Linting
50+
yarn run eslint-full:quiet
51+
```
52+
53+
### Testing
54+
55+
**Backend Tests:**
56+
57+
```bash
58+
# Run all tests
59+
yarn test
60+
61+
# Run specific test file
62+
npx jest tests/actions/action-file-events.test.ts --runInBand --detectOpenHandles --forceExit
63+
```
64+
65+
**CLI/Python Tests:**
66+
67+
```bash
68+
# Run pytest (ensure virtualenv is active)
69+
pytest
70+
```
71+
72+
> [!NOTE]
73+
> See [../docs/development/testing/getting-started.md](../docs/development/testing/getting-started.md) for detailed testing guide.
74+
75+
## 📚 Documentation Reference
76+
77+
- **General Setup**: [../docs/development/getting-started.md](../docs/development/getting-started.md)
78+
- **Testing**: [../docs/development/testing/getting-started.md](../docs/development/testing/getting-started.md)
79+
- **Python/CLI**: [../docs/development/python/getting-started.md](../docs/development/python/getting-started.md)
80+
81+
## 🧪 Test Naming Conventions
82+
83+
To ensure consistency and readability, please adhere to the following naming conventions for test files and test cases:
84+
85+
### File Naming
86+
87+
- **Format**: `kebab-case.test.ts`
88+
- **Location**: `tests/<component>/` or `tests/functional/`
89+
- **Examples**:
90+
- `user-authentication.test.ts`
91+
- `file-upload.test.ts`
92+
- `action-execution.test.ts`
93+
94+
### Test Suite Naming (`describe`)
95+
96+
- **Format**: Title Case or Sentence case describing the component or feature under test.
97+
- **Examples**:
98+
- `describe('User Authentication', ...)`
99+
- `describe('File Upload Service', ...)`
100+
101+
### Test Case Naming (`test` / `it`)
102+
103+
- **Format**: Should start with a verb (e.g., "should", "if") and describe the expected behavior or scenario.
104+
- **Style**:
105+
- **"should" style**: `test('should return 200 OK when valid credentials are provided', ...)`
106+
- **"if" style**: `test('if a user can view details of a submitted action', ...)`
107+
- **Clarity**: Be specific about the condition and the expected result.
108+
109+
### General Rules
110+
111+
- Keep tests focused on a single behavior.
112+
- Use descriptive variable names within tests.
113+
- Clean up resources in `afterEach` or `afterAll` to prevent state leakage.

.github/workflows/test.yml renamed to .github/workflows/api-tests.yml

Lines changed: 41 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -2,65 +2,91 @@ name: Test The Application
22

33
on:
44
pull_request:
5-
types: [opened, synchronize, reopened, edited]
5+
types: [opened, synchronize, reopened]
66
branches:
77
- main
88
- dev
99

1010
jobs:
11-
test_API:
11+
test-api:
1212
runs-on: [self-hosted, linux]
1313

14-
# Permissions reduced since we are no longer posting comments/checks
1514
permissions:
1615
contents: read
1716

1817
steps:
19-
- uses: actions/checkout@v5
18+
- uses: actions/checkout@v6
19+
20+
- uses: pnpm/action-setup@v4
21+
with:
22+
run_install: false
2023

2124
- name: Use Node.js
2225
uses: actions/setup-node@v6
2326
with:
2427
node-version: '22'
25-
package-manager-cache: false
26-
27-
- name: Install yarn
28-
run: npm install -g yarn
28+
cache: 'pnpm'
2929

3030
- name: install dependencies
31-
run: yarn install --immutable
31+
run: pnpm install --frozen-lockfile
32+
33+
- name: Build workspace packages
34+
run: pnpm -r run build
35+
36+
- name: Set up Python
37+
uses: actions/setup-python@v5
38+
with:
39+
python-version: '3.10'
40+
41+
- name: Generate test data
42+
run: |
43+
pip install rosbags
44+
python3 cli/tests/generate_test_data.py
3245
3346
- name: build_stack
3447
run: docker compose -f docker-compose.testing.yml build
48+
env:
49+
BACKEND_URL: 'http://localhost:3000'
50+
GIT_BRANCH: ${{ github.head_ref || github.ref_name }}
51+
GIT_COMMIT: ${{ github.sha }}
3552

3653
- name: clear previous volumes
3754
run: docker compose -f docker-compose.testing.yml down -v
3855

56+
- name: Pre-generate endpoints
57+
working-directory: ./backend
58+
run: |
59+
mkdir -p .endpoints
60+
npm run gen:docs -- .endpoints
61+
3962
- name: launch_stack
4063
working-directory: ./backend
4164
run: |
4265
docker compose -f ../docker-compose.testing.yml up -d
4366
SECONDS=0
4467
TIMEOUT=180
45-
while [ ! -f ./.endpoints/__generated__endpoints.json ]; do
68+
# wait for the api to be reachable
69+
while ! curl -s http://localhost:3000/ > /dev/null; do
4670
if [ $SECONDS -ge $TIMEOUT ]; then
47-
echo "Timeout reached: __generated__endpoints.json not found"
71+
echo "Timeout reached: API not reachable"
72+
docker logs kleinkram-api-server-test
4873
docker compose -f ../docker-compose.testing.yml down
4974
exit 1
5075
fi
5176
sleep 1
52-
echo "waiting for endpoints"
77+
echo "waiting for API to be reachable..."
5378
done
54-
echo "Endpoints found. Running tests..."
79+
80+
echo "API is up. Running tests..."
5581
5682
# This will now cause the step to fail immediately if tests fail.
57-
yarn test --ci
83+
pnpm test --ci
5884
5985
echo "Tests passed successfully."
6086
6187
- name: Clean up
6288
if: always()
63-
run: npm uninstall -g yarn
89+
run: echo "Cleanup done"
6490

6591
- name: Stop Docker stack
6692
if: always()

‎.github/workflows/build-sample-actions.yml‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
name: Build Sample Actions
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
kleinkram_version:
7+
description: 'Kleinkram version to install'
8+
required: false
9+
default: ''
10+
workflow_run:
11+
workflows: ['Update PyPI']
12+
types:
13+
- completed
14+
push:
15+
paths:
16+
- 'examples/kleinkram-actions/**'
17+
18+
jobs:
19+
build-and-push:
20+
runs-on: ubuntu-latest
21+
if: ${{ github.event.workflow_run.conclusion == 'success' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' }}
22+
strategy:
23+
matrix:
24+
action:
25+
[
26+
validate-data,
27+
extract-metadata,
28+
convert-formats,
29+
python-template,
30+
]
31+
type: [dev, prod]
32+
33+
steps:
34+
- name: Checkout code
35+
uses: actions/checkout@v3
36+
37+
- name: Set up Docker Buildx
38+
uses: docker/setup-buildx-action@v2
39+
40+
- name: Login to Docker Hub
41+
uses: docker/login-action@v2
42+
with:
43+
username: ${{ secrets.DOCKERHUB_USERNAME }}
44+
password: ${{ secrets.DOCKERHUB_TOKEN }}
45+
46+
- name: Determine Tag
47+
id: tag
48+
run: |
49+
if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
50+
echo "tag=dev" >> $GITHUB_OUTPUT
51+
echo "kleinkram_version=${{ inputs.kleinkram_version }}" >> $GITHUB_OUTPUT
52+
elif [ "${{ matrix.type }}" == "dev" ]; then
53+
echo "tag=dev" >> $GITHUB_OUTPUT
54+
echo "kleinkram_version=" >> $GITHUB_OUTPUT # Installs latest from PyPI which is updated by Update PyPI workflow
55+
else
56+
echo "tag=latest" >> $GITHUB_OUTPUT
57+
echo "kleinkram_version=" >> $GITHUB_OUTPUT # Installs latest from PyPI
58+
fi
59+
60+
- name: Build and push
61+
uses: docker/build-push-action@v4
62+
with:
63+
context: examples/kleinkram-actions/${{ matrix.action }}
64+
push: true
65+
tags: rslethz/action:${{ matrix.action }}-${{ steps.tag.outputs.tag }}
66+
build-args: |
67+
KLEINKRAM_VERSION=${{ steps.tag.outputs.kleinkram_version }}

‎.github/workflows/check-migrations.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
name: Check Migrations
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- main
7+
8+
jobs:
9+
check-migrations:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- name: Checkout code
13+
uses: actions/checkout@v4
14+
15+
- name: Install pnpm
16+
uses: pnpm/action-setup@v2
17+
with:
18+
version: 10
19+
20+
- name: Setup Node.js
21+
uses: actions/setup-node@v4
22+
with:
23+
node-version: '22'
24+
cache: 'pnpm'
25+
26+
- name: Install dependencies
27+
run: pnpm install --frozen-lockfile
28+
working-directory: backend
29+
30+
- name: Check migrations
31+
run: pnpm run migration:check
32+
working-directory: backend

0 commit comments

Comments
 (0)