Skip to content

Commit f99f343

Browse files
caviricaviri
authored
Merge pull request #4 from Imaging-Plaza/ci-utilities
Merge pull request #3 from Imaging-Plaza/dev
2 parents aaddfae + 25f25a4 commit f99f343

10 files changed

Lines changed: 446 additions & 0 deletions

File tree

‎.devcontainer/Dockerfile‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
FROM ghcr.io/astral-sh/uv:python3.12-bookworm
2+
3+
# Install just and other system dependencies
4+
RUN apt-get update && apt-get install -y \
5+
sudo \
6+
curl \
7+
&& curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin \
8+
&& apt-get clean \
9+
&& rm -rf /var/lib/apt/lists/*
10+
11+
# Crear usuario no-root con UID/GID que suele usar VS Code (1000:1000)
12+
RUN useradd -ms /bin/bash -u 1000 vscode \
13+
&& apt-get update && apt-get install -y sudo \
14+
&& echo "vscode ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
15+
16+
USER vscode
17+
WORKDIR /workspaces

‎.devcontainer/devcontainer.json‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
{
2+
"name": "ai-agent-dev",
3+
"build": {
4+
"dockerfile": "Dockerfile"
5+
},
6+
7+
// This is where your repo will be mounted inside the container
8+
"remoteUser": "vscode",
9+
"workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
10+
11+
"customizations": {
12+
"vscode": {
13+
"settings": {
14+
"python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
15+
"python.envFile": "${workspaceFolder}/.env"
16+
},
17+
"extensions": [
18+
"ms-python.python",
19+
"ms-python.vscode-pylance",
20+
"tamasfe.even-better-toml"
21+
]
22+
}
23+
},
24+
25+
"forwardPorts": [7860],
26+
27+
// Install project in editable mode after the container is built
28+
"postCreateCommand": "rm -rf .venv && uv venv && uv pip install -e . && echo '. $PWD/.venv/bin/activate' >> /home/vscode/.bashrc"
29+
}

‎.github/copilot-instructions.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# AI Agent — Copilot Instructions
2+
3+
This is a **RAG + VLM imaging tool recommender** that helps users find the right imaging software for their images and tasks. Users drop an image, describe their task, and get ranked software recommendations with demo links.
4+
5+
## Architecture Overview
6+
7+
The system follows a two-stage pipeline:
8+
9+
1. **Retrieval Stage** (`retriever/`, `api/pipeline.py`): Fast text search using BGE-M3 embeddings + CrossEncoder reranker. No LLM calls. Returns top-K candidates.
10+
11+
2. **Selection Stage** (`generator/`): Single VLM call (OpenAI GPT-4o/mini) that sees the image + candidates + metadata and returns ranked recommendations with accuracy scores.
12+
13+
### Key Components
14+
15+
- **`api/pipeline.RAGImagingPipeline`**: Main orchestrator. Handles file validation, metadata extraction, retrieval, and VLM selection.
16+
- **`retriever/embedders.py`**: FAISS vector index with BGE-M3 + CrossEncoder reranker for candidate retrieval.
17+
- **`generator/generator.VLMToolSelector`**: Vision-language model that selects best tools from candidates.
18+
- **`utils/image_meta.py`**: Robust metadata extraction for DICOM, NIfTI, TIFF stacks with medical imaging focus.
19+
- **`utils/tags.py`**: Control tags system for query refinement (`[EXCLUDE:tool1|tool2]`, `[NO_RERANK]`, `[REFINE]`).
20+
21+
## Data Flow Patterns
22+
23+
### Input Processing
24+
- Files validated via `utils/file_validator.py` (size limits, format checks)
25+
- Images converted to PNG previews for VLM via `utils/previews.py`
26+
- Metadata extracted preserving original format info (critical for format compatibility matching)
27+
- Format tokens added to retrieval query (e.g. `format:DICOM format:NIfTI`)
28+
29+
### Retrieval Query Construction
30+
```python
31+
# Clean task text + format tokens from uploaded files
32+
query = f"{clean_task} format:{ext_tokens}" # e.g. "segment lungs format:DICOM"
33+
```
34+
35+
### VLM Selection Input
36+
The VLM receives:
37+
- **Text**: User task + candidate table + original file metadata
38+
- **Image**: PNG preview (converted from any format)
39+
- **Metadata**: Original extension, dimensions, file info (crucial for IO compatibility)
40+
41+
## Critical Patterns
42+
43+
### Error Handling
44+
- **Graceful degradation**: If image conversion fails, continue text-only
45+
- **Robust metadata**: All metadata extraction wrapped in try/catch with sensible defaults
46+
- **File validation**: Early validation prevents downstream errors
47+
48+
### Control Tags System
49+
Users can control behavior via tags in their queries:
50+
- `[EXCLUDE:toolname1|toolname2]` - Exclude specific tools from results
51+
- `[NO_RERANK]` - Skip CrossEncoder reranker (faster, less accurate)
52+
- `[REFINE]` - Force clarification turn for alternatives
53+
54+
### Conversation Flow
55+
- **Complete**: Normal success with tool recommendations
56+
- **Needs Clarification**: VLM asks followup questions when task is ambiguous
57+
- **Terminal No-Tool**: No suitable tools found with explanation
58+
59+
## Development Workflows
60+
61+
### Running the App
62+
```bash
63+
# Install with pip using pyproject.toml
64+
pip install -e ".[dev]"
65+
66+
# Configure .env with OPENAI_API_KEY and SOFTWARE_CATALOG path
67+
ai_agent ui # Launches Gradio on port 7860
68+
```
69+
70+
### Testing
71+
- **`tests/full_test.py`**: End-to-end pipeline tests driven by `tests/data/test_data.json`
72+
- Uses test doubles for VLM calls to avoid API costs
73+
- Run with: `pytest tests/`
74+
75+
### Change Documentation
76+
- **`CHANGELOG.md`**: Follow [Keep a Changelog](https://keepachangelog.com/) format
77+
- Use semantic versioning with sections: Added, Changed, Deprecated, Removed, Fixed, Security
78+
- Update CHANGELOG.md for ALL user-facing changes before merging PRs
79+
- Format: `### Added\n- New feature description` under version heading
80+
- Version entries: `## [x.y.z] - YYYY-MM-DD`
81+
82+
### Environment Management
83+
- **uv**: Fast Python package manager used in `tools/image/Dockerfile`
84+
- Creates isolated `.venv` environments for reproducible builds
85+
- Dockerfile uses `uv venv && uv pip install -e .` pattern for container builds
86+
87+
### Logging & Debugging
88+
- Set `LOG_PROMPTS=1` to save VLM prompts + images to `logs/`
89+
- File logs in `logs/app_YYYYMMDD.log` with structured JSON events
90+
- Console/file log levels configurable via `.env`
91+
92+
## Project Conventions
93+
94+
### Schema Patterns
95+
- **Pydantic models** in `generator/schema.py` with robust field validation and aliasing for catalog compatibility
96+
- **Enum-based** conversation states and tool reasons for type safety
97+
- **Field normalization**: Dimensions (2D/3D/4D), modalities (CT/MRI/XR), file formats via validators
98+
99+
### Catalog Integration
100+
- Software catalog in JSONL format following schema.org SoftwareSourceCode structure
101+
- **Runnable examples**: Links to HuggingFace Spaces, notebooks, web demos
102+
- **Supporting data**: Format compatibility info used for matching
103+
104+
### Module Boundaries
105+
- `api/`: Pipeline orchestration, no UI dependencies
106+
- `generator/`: Pure VLM logic, no retrieval dependencies
107+
- `retriever/`: Pure vector search, no generation dependencies
108+
- `utils/`: Shared utilities, no business logic
109+
- `ui/`: Gradio interface only
110+
111+
### Configuration
112+
- Environment-based config via `.env` (API keys, model names, catalog paths)
113+
- Sensible defaults for all settings
114+
- No hardcoded paths or credentials
115+
116+
## Medical Imaging Context
117+
118+
This tool specializes in medical/scientific imaging:
119+
- **Modalities**: CT, MRI, X-ray, Ultrasound, PET, SPECT, Microscopy
120+
- **Formats**: DICOM, NIfTI, TIFF stacks, standard images
121+
- **Dimensions**: 2D images, 3D volumes, 4D timeseries
122+
- **Tasks**: Segmentation, registration, analysis, visualization
123+
124+
The VLM selection considers format compatibility as a primary factor - tools supporting the user's input format are strongly preferred.
125+
126+
## Security Notes
127+
- Only makes external calls to OpenAI VLM API (with user image preview)
128+
- Never uploads user data to third-party tool demos
129+
- Returns links only; user chooses whether to visit demos
130+
- Prompt logging is optional and local-only

‎.github/workflows/publish_image_in_GHCR.yaml‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
name: Build and Publish Docker Images
2+
3+
on:
4+
push:
5+
branches: [ "main", "develop" ]
6+
pull_request:
7+
branches: [ "main", "develop" ]
8+
9+
jobs:
10+
build-and-publish:
11+
runs-on: ubuntu-latest
12+
permissions:
13+
contents: write # needed to create the release
14+
packages: write # needed to publish the image
15+
16+
# Skip building images for draft PRs
17+
if: |
18+
github.event_name == 'push' ||
19+
(github.event_name == 'pull_request' &&
20+
github.event.pull_request.draft == false)
21+
22+
steps:
23+
- name: Checkout repository
24+
uses: actions/checkout@v4
25+
26+
- name: Extract version from pyproject.toml
27+
id: project_version
28+
run: |
29+
VERSION=$(grep 'version =' pyproject.toml | sed -E 's/version = "([^"]+)"/\1/')
30+
echo "version=${VERSION}" >> $GITHUB_OUTPUT
31+
32+
- name: Extract changelog section for version
33+
id: changelog
34+
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
35+
run: |
36+
VERSION="${{ steps.project_version.outputs.version }}"
37+
38+
# Extract the section for this version from CHANGELOG.md
39+
# This awk script finds the section between [VERSION] and the next [VERSION] or end of file
40+
CHANGELOG_SECTION=$(awk -v version="[$VERSION]" '
41+
BEGIN { found=0; content="" }
42+
$0 ~ "^## \\[" {
43+
if (found) exit
44+
if ($0 ~ version) {
45+
found=1
46+
content = $0 "\n"
47+
next
48+
}
49+
}
50+
found { content = content $0 "\n" }
51+
END { print content }
52+
' CHANGELOG.md)
53+
54+
# If no section found, use a default message
55+
if [ -z "$CHANGELOG_SECTION" ]; then
56+
CHANGELOG_SECTION="## Release v${VERSION}\n\nNo changelog entry found for this version."
57+
fi
58+
59+
# Save to file and output
60+
echo "$CHANGELOG_SECTION" > release_notes.md
61+
echo "changelog_file=release_notes.md" >> $GITHUB_OUTPUT
62+
63+
# Also output as multiline string for debugging
64+
{
65+
echo 'content<<EOF'
66+
echo "$CHANGELOG_SECTION"
67+
echo 'EOF'
68+
} >> $GITHUB_OUTPUT
69+
70+
- name: Log in to GitHub Container Registry
71+
uses: docker/login-action@v3
72+
with:
73+
registry: ghcr.io
74+
username: ${{ github.actor }}
75+
password: ${{ secrets.GITHUB_TOKEN }}
76+
77+
- name: Extract metadata for Docker
78+
id: meta
79+
uses: docker/metadata-action@v5
80+
with:
81+
images: ghcr.io/${{ github.repository }}
82+
tags: |
83+
# For main branch: latest and version tags
84+
type=raw,value=latest,enable={{is_default_branch}}
85+
type=raw,value=${{ steps.project_version.outputs.version }},enable={{is_default_branch}}
86+
# For develop branch: develop tag
87+
type=raw,value=develop,enable=${{ github.ref == 'refs/heads/develop' }}
88+
# For PRs only: pr-{number} tag
89+
type=ref,event=pr,prefix=pr-
90+
labels: |
91+
org.opencontainers.image.title=${{ github.repository }}
92+
org.opencontainers.image.description=${{ github.event.repository.description }}
93+
org.opencontainers.image.url=${{ github.event.repository.html_url }}
94+
org.opencontainers.image.source=${{ github.event.repository.clone_url }}
95+
org.opencontainers.image.revision=${{ github.sha }}
96+
org.opencontainers.image.licenses=${{ github.event.repository.license.spdx_id }}
97+
# Add cleanup hint for PR images
98+
io.github.pr-image=${{ github.event_name == 'pull_request' && 'true' || 'false' }}
99+
100+
- name: Build and push Docker image
101+
uses: docker/build-push-action@v5
102+
with:
103+
context: .
104+
file: tools/image/Dockerfile
105+
platforms: linux/amd64
106+
push: true
107+
tags: ${{ steps.meta.outputs.tags }}
108+
labels: ${{ steps.meta.outputs.labels }}
109+
110+
- name: Create GitHub Release
111+
# Only create releases for main branch pushes
112+
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
113+
uses: softprops/action-gh-release@v2
114+
with:
115+
tag_name: v${{ steps.project_version.outputs.version }}
116+
name: Release v${{ steps.project_version.outputs.version }}
117+
body_path: ${{ steps.changelog.outputs.changelog_file }}
118+
fail_on_unmatched_files: false
119+
120+
# Clean up PR images when PR is closed
121+
cleanup-pr-image:
122+
runs-on: ubuntu-latest
123+
if: github.event_name == 'pull_request' && github.event.action == 'closed'
124+
permissions:
125+
packages: write
126+
127+
steps:
128+
- name: Delete PR image
129+
uses: actions/github-script@v7
130+
with:
131+
github-token: ${{ secrets.GITHUB_TOKEN }}
132+
script: |
133+
const owner = context.repo.owner;
134+
const repo = context.repo.repo;
135+
const packageName = `${owner}/${repo}`;
136+
const prNumber = context.payload.pull_request.number;
137+
const prTag = `pr-${prNumber}`;
138+
139+
try {
140+
// Get all package versions
141+
const { data: versions } = await github.rest.packages.getAllPackageVersionsForPackageOwnedByOrg({
142+
package_type: 'container',
143+
package_name: packageName,
144+
org: owner,
145+
per_page: 100
146+
});
147+
148+
// Find the PR image version
149+
const prVersion = versions.find(version =>
150+
version.metadata.container.tags.includes(prTag)
151+
);
152+
153+
if (prVersion) {
154+
console.log(`Deleting PR image: ${prTag} (version ID: ${prVersion.id})`);
155+
await github.rest.packages.deletePackageVersionForOrg({
156+
package_type: 'container',
157+
package_name: packageName,
158+
org: owner,
159+
package_version_id: prVersion.id
160+
});
161+
console.log(`Successfully deleted PR image: ${prTag}`);
162+
} else {
163+
console.log(`No image found for PR: ${prTag}`);
164+
}
165+
} catch (error) {
166+
console.log(`Error cleaning up PR image (this is normal if no image was built): ${error.message}`);
167+
}

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Changelog
2+
3+
All notable changes to this project will be documented in this file.
4+
5+
## [0.1.0] - 2025-09-30
6+
7+
### Added
8+
- Chat functionality

0 commit comments

Comments
 (0)