Skip to content

AllenNeuralDynamics/aind-metadata-viz

BranchesTags

Repository files navigation

AIND Metadata Portal

metadata visualizations

REST API

The portal exposes a REST API served by FastAPI/uvicorn.

Gather endpoint

Gathers and validates metadata from the metadata service for a given subject.

  • POST /gather — body is a JSON object with required subject_id and project_name
  • Optional body keys: metadata_service_url, modalities (list), tags (list), group, restrictions, data_summary, acquisition_start_time
response = requests.post(
    "https://metadata-portal.allenneuraldynamics-test.org/gather",
    json={"subject_id": "123456", "project_name": "MyProject"},
)
print(response.json())

Upgrade endpoint

POST /upgrade accepts a metadata.json dict and runs it through aind-metadata-upgrader. It always returns original and upgraded JSON side by side for each field, even when some fields fail.

import requests, json

with open("metadata.json") as f:
    metadata = json.load(f)

response = requests.post(
    "https://metadata-portal.allenneuraldynamics.org/upgrade",
    json=metadata,
)
result = response.json()
# result["overall_success"]  — True if all fields upgraded cleanly
# result["overall_error"]    — error string if the full upgrade failed
# result["partial_success"]  — True if at least one field succeeded
# result["files_tested"]     — per-field breakdown

Query endpoints

  • GET /upgrade-query — Build a query using the LLM query builder.
  • POST /retrieve-records — Run a query against the metadata store.

Optional query parameters for /retrieve-records:

  • names_only=true — return only asset names
  • limit=<int> — limit number of results (default 0 = no limit)
  • projection=<json> — JSON object specifying which fields to include/exclude
response = requests.post(
    "https://metadata-portal.allenneuraldynamics-test.org/retrieve-records",
    params={"limit": 10, "projection": '{"subject.subject_id": 1, "name": 1}'},
    json={"subject.subject_id": "123456"},
)
print(response.json())

Chat endpoint

POST /chat — Ask a natural-language question about the metadata store. The agent can query records, look up schema info, and summarize results.

Query parameters:

Parameter Type Required Description
id string no Optional caller identifier logged with each request (e.g. a username or app name)

Request body:

Field Type Required Description
message string yes User question (max 4096 bytes)
history list no Prior turns; each {"role": "user"|"assistant", "content": "..."} (max 20 turns)

Response body:

Field Description
response Agent's answer text
stop_reason Why the agent stopped (e.g. "end_turn")
iterations Number of reasoning steps
tool_calls List of {name, input, output, is_error} tool invocations

Rate-limited per IP (default: 10/min, 200/day).

Every successful request is appended as a JSON Lines record to a daily log file in S3: s3://aind-scratch-data/aind-metadata-viz-logs/chat_log_{YYYY-MM-DD}.json

Each log record contains:

Field Description
timestamp ISO-8601 UTC timestamp
requester_id Value of the ?id= query parameter, or null
ip Client IP address
message User message
response Agent response
stop_reason Agent stop reason
iterations Number of agent reasoning steps
tool_call_count Number of tool calls made 
response = requests.post(
    "https://metadata-portal.allenneuraldynamics.org/chat",
    params={"id": "my-app"},
    json={"message": "How many SmartSPIM assets are there?"},
)
print(response.json()["response"])

Contributions endpoints

Stores and retrieves CRediT authorship contributions for a project, versioned via SQLite.

Endpoint Description
GET /contributions/get?project=<name> Latest contribution data (JSON by default)
GET /contributions/get?project=<name>&format=yaml Latest contribution data as YAML
GET /contributions/get?project=<name>&commit=<hash> Contribution data at a specific commit
GET /contributions/get?project=<name>&history=true List of all commits, newest first
GET /contributions/get?doi=<doi> Look up a project by DOI
POST /contributions/post?project=<name>[&message=<msg>][&password=<hash>] Store a new version; body is JSON or YAML
GET /contributions/token?doi=<doi>&type=add_author|edit_author[&author=<name>][&days=<n>][&password=<hash>] Create a scoped token for a project
GET /contributions/author-image?author=<name> Returns the S3 key for an author's headshot

Local dev

uv sync --extra dev
uvicorn aind_metadata_viz.main:app --reload

CI/CD

docker build -t aind-metadata-viz .
docker run -p 8000:8000 aind-metadata-viz

The metadata portal hosts validation endpoints for the latest aind-data-schema release. You can hit these endpoints with:

Example

import requests
import json

with open("metadata.json", "r") as f:
    metadata = json.load(f)

response = requests.post(
    "https://metadata-portal.allenneuraldynamics-test.org/validate/metadata", 
    json=metadata
)

if response.status_code == 200:
    print("✅ Validation passed!")
else:
    print(f"❌ Validation failed: {response.json()}")

Files endpoint

You can also post a dictionary containing only the core files (i.e. not generated from a Metadata object). This can be useful when testing whether your metadata are valid prior to triggering the aind-data-transfer-service on a data asset.

import requests
import json

core_files = ["data_description", "acquisition", "instrument", "procedures", "subject"]

metadata = {}
for core_file in core_files:
    with open(f"{core_file}.json", "r") as f:
        metadata[core_file] = json.load(f)

response = requests.post(
    "https://metadata-portal.allenneuraldynamics-test.org/validate/files", 
    json=metadata
)

if response.status_code == 200:
    print("✅ Validation passed!")
else:
    print(f"❌ Validation failed: {response.json()}")

Individual validation endpoints

  • /validate/subject - Subject metadata
  • /validate/data_description - Data description metadata
  • /validate/acquisition - Acquisition metadata
  • /validate/instrument - Instrument metadata
  • /validate/procedures - Procedures metadata
  • /validate/processing - Processing metadata
  • /validate/quality_control - Quality control metadata
  • /validate/model - Model metadata

Example usage: requests.post("https://metadata-portal.allenneuraldynamics-test.org/validate/subject", json=subject_data)

Gather endpoint

Gathers and validates metadata from the metadata service for a given subject.

  • GET /gather?subject_id=<id>&project_name=<name> — required parameters
  • Optional: metadata_service_url, modalities (comma-separated), tags (comma-separated), group, restrictions, data_summary, acquisition_start_time
response = requests.get(
    "https://metadata-portal.allenneuraldynamics-test.org/gather",
    params={"subject_id": "123456", "project_name": "MyProject"},
)
print(response.json())

Upgrade endpoint

POST /upgrade accepts a metadata.json dict and runs it through aind-metadata-upgrader. It always returns original and upgraded JSON side by side for each field, even when some fields fail.

import requests, json

with open("metadata.json") as f:
    metadata = json.load(f)

response = requests.post(
    "https://metadata-portal.allenneuraldynamics.org/upgrade",
    json=metadata,
)
result = response.json()
# result["overall_success"]  — True if all fields upgraded cleanly
# result["overall_error"]    — error string if the full upgrade failed
# result["partial_success"]  — True if at least one field succeeded
# result["files_tested"]     — per-field breakdown:
#   {
#     "subject": {"success": True, "error": None, "original": {...}, "upgraded": {...}},
#     "acquisition": {"success": False, "error": "...", "original": {...}, "upgraded": null},
#     ...
#   }

Fields that rename across schema versions (e.g. session → acquisition, rig → instrument) include a "converted_to" key indicating the new name.

Query endpoints

  • GET /upgrade-query — Build a query using the LLM query builder. Pass query string parameters as needed.
  • POST /retrieve-records — Run a query. Accepts a JSON object as the request body.

Optional query parameters for /retrieve-records:

  • names_only=true — return only asset names
  • limit=<int> — limit number of results (default 0 = no limit)
  • projection=<json> — JSON object specifying which fields to include/exclude (e.g. {"subject.subject_id": 1})
response = requests.post(
    "https://metadata-portal.allenneuraldynamics-test.org/retrieve-records",
    params={"limit": 10, "projection": '{"subject.subject_id": 1, "name": 1}'},
    json={"subject.subject_id": "123456"},
)
print(response.json())

Chat endpoint

POST /chat — Ask a natural-language question about the metadata store. The agent can query records, look up schema info, and summarize results.

Query parameters:

Parameter Type Required Description
id string no Optional caller identifier logged with each request (e.g. a username or app name)

Request body:

Field Type Required Description
message string yes User question (max 4096 bytes)
history list no Prior turns; each {"role": "user"|"assistant", "content": "..."} (max 20 turns)

Response body:

Field Description
response Agent's answer text
stop_reason Why the agent stopped (e.g. "end_turn")
iterations Number of reasoning steps
tool_calls List of {name, input, output, is_error} tool invocations

Rate-limited per IP (default: 10/min, 200/day).

Every successful request is appended as a JSON Lines record to a daily log file in S3: s3://aind-scratch-data/aind-metadata-viz-logs/chat_log_{YYYY-MM-DD}.json

Each log record contains:

Field Description
timestamp ISO-8601 UTC timestamp
requester_id Value of the ?id= query parameter, or null
ip Client IP address
message User message
response Agent response
stop_reason Agent stop reason
iterations Number of agent reasoning steps
tool_call_count Number of tool calls made 
response = requests.post(
    "https://metadata-portal.allenneuraldynamics.org/chat",
    params={"id": "my-app"},
    json={"message": "How many SmartSPIM assets are there?"},
)
print(response.json()["response"])

Contributions endpoints

Stores and retrieves CRediT authorship contributions for a project, versioned via SQLite.

Endpoint Description
GET /contributions/get?project=<name> Latest contribution data (JSON by default)
GET /contributions/get?project=<name>&format=yaml Latest contribution data as YAML
GET /contributions/get?project=<name>&commit=<hash> Contribution data at a specific commit
GET /contributions/get?project=<name>&history=true List of all commits, newest first
GET /contributions/get?doi=<doi> Look up a project by DOI
POST /contributions/post?project=<name>[&message=<msg>][&password=<hash>] Store a new version; body is JSON or YAML
GET /contributions/token?doi=<doi>&type=add_author|edit_author[&author=<name>][&days=<n>][&password=<hash>] Create a scoped one-time (add_author) or reusable (edit_author) token for a project
GET /contributions/author-image?author=<name> Returns {"author": "<name>", "image_key": "<s3-key>"} for the author's headshot; 404 if not found

All models are publicly readable — GET requests never require a password. To lock a project, supply ?password=<hash> on a POST to an unlocked project; that hash becomes the required password for all future saves. To re-save a locked project, supply the same ?password=<hash>; omitting or supplying the wrong value returns 401.

Pull the seeded IBL example to see a complete payload with all fields:

import requests, json

r = requests.get(
    "https://metadata-portal.allenneuraldynamics-test.org/contributions/get",
    params={"project": "ibl-2025"},
)
print(json.dumps(r.json(), indent=2))

Fetch the full history for a project and retrieve a specific past version:

import requests, json

history = requests.get(
    "https://metadata-portal.allenneuraldynamics-test.org/contributions/get",
    params={"project": "MyProject", "history": "true"},
).json()

oldest_commit = history[-1]["commit"]
old_version = requests.get(
    "https://metadata-portal.allenneuraldynamics-test.org/contributions/get",
    params={"project": "MyProject", "commit": oldest_commit},
).json()

Usage

Clone the repository and cd into the folder

Create a virtual environment and install the package, then launch panel.

python -m venv .venv
source .venv/bin/activate (or .venv/bin/Scripts/activate on Windows)
pip install -e .
panel serve ./src/aind_metadata_viz/app.py --show --autoreload

To launch the query and view apps replace the app.py with the appropriate launcher.

CI/CD

There is a Dockerfile which includes the entrypoint to launch the app.

Local dev

  1. Build the Docker image locally and run a Docker container:
docker build -t aind-metadata-viz .
docker run -e ALLOW_WEBSOCKET_ORIGIN=localhost:8000 -p 8000:8000 aind-metadata-viz
  1. Navigate to 'localhost:8000` to view the app.

AWS

  1. On pushes to the dev or main branch, a GitHub Action will run to publish a Docker image to ghcr.io/allenneuraldynamics/aind-metadata-viz:dev or ghcr.io/allenneuraldynamics/aind-metadata-viz:latest.
  2. The image can be used by a ECS Service in AWS to run a task container. Application Load Balancer can be used to serve the container from ECS. Please note that the task must be configured with the correct env variables (e.g. API_GATEWAY_HOST, ALLOW_WEBSOCKET_ORIGIN).

About

Website for public metadata visualization

Resources

Readme

License

MIT license
Activity
Custom properties

Stars

0 stars

Watchers

5 watching

Forks

1 fork
Report repository

Releases

74 tags

Packages

 
 
 

Contributors

Languages