Skip to content

feat: add initial eval scripts for unified memory#1106

Draft
maxervo wants to merge 5 commits into
developfrom
feat/vss-unified-memory
Draft

feat: add initial eval scripts for unified memory#1106
maxervo wants to merge 5 commits into
developfrom
feat/vss-unified-memory

Conversation

@maxervo

@maxervo maxervo commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Description

Eval scripts to get started on automating evaluating unified memory with scenarios/questions from body cam videos.
Iterated upon as we are running tests.

Drive the eval deterministically with context isolation for eval purity.

WIP in this draft:

  • Code/folders being restructured in this draft
  • Preparing instructions to run and reproduce results
  • JSON standardization, better reporting etc.

Next steps after this PR:

  • Concurrent tasks for independent run. Specify option for one run. Faster feedback
  • Modularize code to cover both single/cross evals. Design abstractions and reduce duplicate code
  • Better abstraction: decouple agent harness being tested so we can drive it with others (hermes, existing vss agent interactions)
  • Refactor with existing VSS utilities, maybe NAT eval etc. (For now, just get it working with basics/simple python to get started and get initial results)

Checklist

  • I am familiar with the Contributing Guidelines.
  • I have installed and run pre-commit hooks locally (uv run pre-commit install once, then hooks run on every git commit).
  • Every commit on this PR is DCO sign-off'd (git commit -s adds a Signed-off-by trailer that certifies you have the right to submit the change under Apache-2.0).
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Signed-off-by: Maxime Alardo <malardo@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jun 25, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Maxime Alardo <malardo@nvidia.com>
@maxervo maxervo force-pushed the feat/vss-unified-memory branch from 3731540 to 4059c98 Compare June 25, 2026 20:54
Signed-off-by: Maxime Alardo <malardo@nvidia.com>
@maxervo maxervo force-pushed the feat/vss-unified-memory branch from fa4ec7a to 12aac3a Compare June 25, 2026 22:19
@maxervo maxervo force-pushed the feat/vss-unified-memory branch from 4ad45aa to 2d61803 Compare June 25, 2026 22:58
@bp-sre-sonarqube

Copy link
Copy Markdown

Quality Gate passed Quality Gate passed for 'video-search-and-summarization-skills'

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@bp-sre-sonarqube

Copy link
Copy Markdown

Quality Gate passed Quality Gate passed for 'video-search-and-summarization-ui'

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@bp-sre-sonarqube

Copy link
Copy Markdown

Quality Gate passed Quality Gate passed for 'video-search-and-summarization-agent'

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant