Skip to content

Darkfish_Foundry_Demo_Notebook_Writeup.md

Latest commit

 

History

History
492 lines (338 loc) · 14.7 KB

Darkfish_Foundry_Demo_Notebook_Writeup.md

File metadata and controls

492 lines (338 loc) · 14.7 KB
Raw

Darkfish Foundry Demo Notebook Write-Up

Project: Darkfish Foundry Demo
Demo package: v0.8.1.2
Runtime package version: 0.8.1-1
Submission type: MVP demo application write-up
Primary surfaces: Windows desktop GUI, local agent CLI, local STDIO protocol

1. Executive Summary

Darkfish Foundry is a local-first source intake and artifact-generation tool. The current submission is a demo of the MVP: it shows a working desktop application and agent-callable command line interface for turning video transcript sources, local documents, and native-text PDFs into structured local artifacts.

The demo focuses on a practical problem: researchers, builders, analysts, and agents often need to collect source material, normalize it, preserve provenance, and generate reusable text/metadata artifacts without sending local files to cloud services. Darkfish Foundry provides a small but functional foundation for that workflow.

This demo is not positioned as a complete production platform. It is a working MVP that demonstrates the core architecture, local-first privacy boundary, user GUI, agent CLI, native-text PDF lane, transcript lane, document lane, artifact outputs, and safe agent-facing behavior.

2. What The Demo Does

Darkfish Foundry Demo can:

  • Launch as a Windows desktop application through Darkfish-Foundry.exe.
  • Process video transcript URLs into local transcript artifacts.
  • Process local documents into normalized output formats.
  • Extract native text from PDFs through a packaged PDFium runtime.
  • Export artifacts as Markdown, plain text, JSON, YAML, DOCX, and PDF extraction bundles where supported.
  • Maintain local history and artifact records.
  • Provide an agent-callable CLI for automation without opening the GUI.
  • Provide a line-oriented STDIO protocol for external agents.
  • Validate runtime readiness with machine-readable JSON checks.
  • Keep local documents and PDFs on the local machine.

The demo includes a cleaned user-facing package. Source/developer scaffolding, old smoke reports, and local development paths have been removed from the v0.8.1.2 package.

3. Intended Users

Darkfish Foundry is intended for:

  • Researchers collecting and normalizing source material.
  • Analysts preparing local evidence bundles.
  • Builders creating repeatable source-intake workflows.
  • Agents that need a stable, local command interface for document and transcript processing.
  • Users who want a desktop GUI but also need headless automation.

4. Local-First Privacy Boundary

Darkfish Foundry is local-first and has no telemetry in this demo.

Source Type Network Behavior
Local text/document files Stay local
Native-text PDFs Stay local
Video transcript URLs Contact the video destination only for transcript retrieval
Generated artifacts Written to the selected local output folder

Original source files are read, not modified.

5. Application Architecture

At a high level, the demo has four layers:

  1. Desktop GUI
    A Tauri/Svelte desktop interface for human users.

  2. Shared Rust Core
    Core processing logic for transcript jobs, document jobs, PDF extraction, artifact writing, history, and validation.

  3. Agent CLI And STDIO Surface
    A local darkfish.exe command interface designed for external agents or scripts.

  4. Packaged Runtime Sidecars
    Local helper binaries and runtime libraries, including yt-dlp for transcript retrieval and PDFium for native-text PDF extraction.

User GUI ─┐
          ├── Shared Rust Core ── Artifacts + SQLite records
Agent CLI ┘            │
                       ├── yt-dlp sidecar for transcript jobs
                       └── PDFium runtime for native-text PDF jobs

6. Package Layout

The cleaned demo package keeps only user-facing runtime files and documentation.

Darkfish-Foundry.exe
README.md
DEPLOYMENT-README.md
RELEASE-MANIFEST.md
THIRD_PARTY_NOTICES.md

docs/
  agent-cli.md
  agent-stdio-protocol.md
  darkfish-artifact-result-schema.md
  skills/darkfish-foundry-agent-cli/SKILL.md

release/windows/
  darkfish-foundry.exe
  darkfish.exe
  Darkfish Foundry_0.8.1-1_x64-setup.exe
  Darkfish Foundry_0.8.1-1_x64_en-US.msi
  THIRD_PARTY_NOTICES.md
  sidecars/
    yt-dlp.exe
    yt-dlp-x86_64-pc-windows-msvc.exe
    pdfium/pdfium.dll
    pdfium/licenses/

7. How To Launch The Desktop Demo

Code Cell: Start The GUI

# From the demo package root:
.\Darkfish-Foundry.exe

The launcher validates that the portable runtime bundle exists, then starts the packaged desktop GUI:

release\windows\darkfish-foundry.exe

8. Agent CLI Health Check

Darkfish provides a machine-readable health check for agents and scripts.

Code Cell: Verify Runtime Readiness

.\release\windows\darkfish.exe doctor --json

Expected behavior:

{
  "ok": true,
  "status": "succeeded",
  "data": {
    "darkfish_version": "0.8.1-1",
    "sidecar_found": true,
    "output_writable": true,
    "warnings": []
  }
}

The exact paths depend on the active Windows user profile and chosen output directory.

9. Transcript Processing

The transcript lane takes a video URL and writes local transcript artifacts. It is designed for caption/transcript retrieval, not video media download.

Code Cell: Process A Transcript URL

.\release\windows\darkfish.exe process transcript "https://www.youtube.com/watch?v=VIDEO_ID" --format txt --format md --format srt --json

Outputs may include:

transcript.txt
transcript.md
transcript.srt
manifest/history records

Potential external failure states include unavailable captions, private videos, rate limits, network errors, auth gates, or extractor changes. Those are reported as source/external failures rather than local document-processing failures.

10. Local Document Processing

The document lane processes local files without network access.

Supported local input types in the MVP demo:

txt, md, markdown, json, yaml, yml, csv, log, native-text pdf

Supported output formats:

txt, md, json, yaml, docx

Code Cell: Process A Single Document

.\release\windows\darkfish.exe process document "C:\path\source.txt" --format md --format json --format yaml --format docx --metadata lite --json

The output is written to a local artifact folder under the selected output directory.

11. Lite Metadata

The demo includes a lightweight deterministic metadata layer for local documents.

When --metadata lite is used, Darkfish writes:

metadata.json

The current metadata profile is:

darkfish.usas_lite.v0_2

The document envelope is:

darkfish.document.v0_8_1_1.provisional

This metadata is intentionally conservative. It is not full semantic extraction, not a source-truth certification layer, and not an LLM interpretation layer. It records source identity, hashes, extraction details, simple evidence anchors, deterministic key-term candidates, safety flags, and routing guidance.

Unsupported metadata modes fail clearly with unsupported_metadata_mode; the tool does not silently fall back to a different schema.

12. PDF Native-Text Extraction

The PDF lane uses a packaged PDFium runtime for native-text PDFs.

Code Cell: Check PDF Runtime

.\release\windows\darkfish.exe pdf capabilities --json

Code Cell: Preflight A PDF

.\release\windows\darkfish.exe pdf preflight "C:\path\file.pdf" --json

Preflight checks:

  • PDF extension and magic/header validation.
  • PDFium runtime availability.
  • Page count.
  • Native text availability.
  • Security markers such as JavaScript, launch actions, embedded files, and XFA markers.

Code Cell: Extract Native Text From A PDF

.\release\windows\darkfish.exe pdf extract "C:\path\file.pdf" --output "C:\path\out" --json

Successful PDF extraction writes:

canonical.txt
document.md
layout.json
quality_report.json
processing_journal.json
manifest.json

Code Cell: Verify A PDF Bundle

.\release\windows\darkfish.exe pdf verify "C:\path\out\PDF_Documents\file_hash" --json

Verification checks generated bundle files against manifest hashes.

13. Queue And Batch Operations

Darkfish supports queue workflows for agents and batch jobs.

Code Cell: Queue A Document

.\release\windows\darkfish.exe queue add --source-kind document "C:\path\source.txt" --format md --format json --json
.\release\windows\darkfish.exe queue run --once --json

Code Cell: Queue A PDF

.\release\windows\darkfish.exe queue add --source-kind pdf "C:\path\source.pdf" --format txt --format md --json
.\release\windows\darkfish.exe queue run --once --json

Code Cell: Queue From A File List

.\release\windows\darkfish.exe queue import --source-kind document file-paths.txt --format md --format json --json
.\release\windows\darkfish.exe queue run --drain --json

A mixed batch may return partial if some files succeed and others fail or are unsupported. The GUI uses the simpler partial status label with review-warning context.

14. Agent STDIO Protocol

Darkfish exposes a JSON-lines STDIO interface for agents.

Code Cell: Start STDIO Mode

.\release\windows\darkfish.exe agent --stdio

Request shape:

{"id":"h1","method":"health.check","params":{}}

Success response shape:

{"id":"h1","ok":true,"status":"succeeded","result":{}}

Failure response shape:

{"id":"x1","ok":false,"status":"failed","error":{"code":"invalid_args","message":"...","stage":"args","retryable":false}}

STDIO methods include:

health.check
process.transcript
process.document
process.documents
pdf.capabilities
pdf.preflight
pdf.extract
pdf.ingest
pdf.verify
queue.add
queue.import
queue.list
queue.status
queue.run_once
queue.run_drain
artifact.path
artifact.get

15. Artifact Output Philosophy

Darkfish does not treat command success alone as enough. Agent-facing workflows should verify generated artifacts.

A safe agent trajectory is:

  1. Identify source type and scope.
  2. Ask or confirm output destination for persistent jobs.
  3. Run doctor --json.
  4. Run PDF capabilities/preflight for PDF work.
  5. Process the smallest scoped job that matches the request.
  6. Parse JSON output only.
  7. Verify artifact paths, sizes, hashes, and existence where returned.
  8. Retrieve or inspect at least one artifact before reporting success.
  9. Report warnings, partial results, and external failures honestly.

16. Limitations Of This MVP Demo

This is a demo of the MVP, not the complete final product.

Known limitations:

  • Windows-only packaged demo at this stage.
  • PDF support is native-text extraction only; OCR is not included.
  • Scanned/image-only PDFs return ocr_required.
  • Transcript jobs depend on video destination availability, captions, rate limits, and extractor behavior.
  • The demo does not download video media.
  • The metadata profile is lightweight and deterministic, not full semantic extraction.
  • DOCX is currently an output format; rich DOCX input parsing is not the focus of this demo.
  • Folder intake is intentionally bounded and non-recursive.
  • The GUI is functional but still MVP-level.
  • Installer signing, auto-updates, and enterprise deployment polish are not included.
  • The package version is 0.8.1-1 internally because Cargo/Tauri/MSI require SemVer-compatible versioning, while the cleaned demo folder is labeled v0.8.1.2.

17. Planned Additions

Planned future work includes:

  • OCR support for scanned PDFs and image-based documents.
  • Stronger PDF security review and richer PDF layout extraction.
  • Expanded document input support, including richer DOCX parsing.
  • More polished installer and code-signing workflow.
  • Better queue management and batch progress UX in the GUI.
  • More robust playlist and multi-source transcript workflows.
  • Additional metadata profiles after schema review.
  • Optional validation reports suitable for research/evidence workflows.
  • Cross-platform packaging after the Windows MVP stabilizes.
  • More formal regression test packs for agent CLI and STDIO behavior.
  • A polished splash animation and demo video showcase.

18. Why This Matters For Agentic Workflows

The core design goal is not only that a human can click a button. The goal is also that an agent can safely operate the same processing capabilities through a stable local interface.

Darkfish Foundry demonstrates:

  • A real local executable that agents can call directly.
  • Machine-readable success and failure envelopes.
  • Local artifact creation and retrieval.
  • Explicit failure codes for unsupported inputs and unavailable runtime dependencies.
  • Clear boundaries between local processing and external video transcript retrieval.
  • Documentation that tells agents what safe behavior looks like.

This makes the demo useful as a foundation for agentic source intake, evidence preparation, and local artifact generation.

19. Competition Submission Framing

Darkfish Foundry Demo should be evaluated as a working MVP demonstration of a local-first source processing system with both human and agent interfaces.

The central claim is:

A user or local agent can launch Darkfish Foundry, verify runtime readiness, process transcript/document/PDF sources, generate local artifacts, and inspect outputs without relying on cloud processing for local files.

The project is intentionally scoped. It prioritizes privacy boundaries, local artifacts, agent-readability, and safe operational contracts over broad feature expansion.

20. Minimal Evaluator Checklist

An evaluator can inspect or run the demo as follows:

# 1. Launch the GUI
.\Darkfish-Foundry.exe

# 2. Verify CLI readiness
.\release\windows\darkfish.exe doctor --json

# 3. Verify PDF runtime
.\release\windows\darkfish.exe pdf capabilities --json

# 4. Process a local text document
.\release\windows\darkfish.exe process document "C:\path\source.txt" --format md --format json --metadata lite --json

# 5. Start agent STDIO mode
.\release\windows\darkfish.exe agent --stdio

For local document and PDF jobs, source files stay on the evaluator's machine. For transcript jobs, the video destination is contacted only when a transcript job is explicitly run.

21. Closing Note

Darkfish Foundry Demo is an MVP with a working GUI, a packaged local CLI, PDF native-text extraction, transcript processing, document artifact generation, local history, and agent-facing contracts. It is designed to show the feasibility of a local-first source processing tool that can be used by both humans and agents while keeping source material, outputs, and operational boundaries explicit.