Add finetune MCP server with real training path, mock toggle, templates and CLI demo

[agus-official]

Motivation

• Provide a real finetune execution path (Transformers + Datasets + PEFT) alongside the existing mock/demo flow so demos can be toggled into real training mode.
• Expose finetune primitives (model listing, dataset search/download, synthetic generation, template preparation, async job lifecycle and evaluation) as MCP tools for deterministic workflow composition.
• Allow workflows and the CentralAgent/NodeAgent to discover and use tools from multiple MCP servers and make the finetune pipeline runnable from the CLI without external orchestration.
• Support real Hugging Face datasets/repos and a configurable real functiongemma repo while keeping a safe mock fallback for demos.

Description

• Added a new MCP server package mcp_servers/finetune_server implementing server.py (tool handlers), job_store.py (SQLite-backed job/events store), and __main__.py for stdio transport, plus a JSON-stable helper and _normalize_model_id that reads FUNCTIONGEMMA_HF_REPO from env.
• Added a real training template under apps/finetune_workflow/templates/base with train.py (uses transformers, datasets, peft) and config.json, and a scripts/run_finetune_demo.py CLI that wires mock_mode into the sandbox run input.
• Integrated the finetune server into the runtime by registering it in core/mcp/config.py, adding MCPServerManager.list_server_ids(), and changing NodeAgent to aggregate MCP tool catalogs across all servers; also added API endpoints /v1/finetune/jobs/{job_id} and /v1/finetune/jobs/{job_id}/events in apps/api/main.py.
• Added workflow and agent assets (data/agent_specs/finetune_planner.json, data/agent_specs/finetune_monitor.json, data/schemas/finetune_*.json, data/workflow_archetypes/finetune_llm_auto_v1.json, and a concrete workflow), updated requirements.txt with training-related libs, and added tests/test_finetune_mcp.py exercising the mock lifecycle.

Testing

• Added an automated test tests/test_finetune_mcp.py that exercises the mocked finetune lifecycle (list_models -> search_datasets -> download_dataset -> prepare_template -> start_job -> wait_for_job -> run_evaluation).
• The new test file was added to the repository but no automated test suite was executed as part of this change.
• No CI or unit test runs were performed here, so there are no recorded pass/fail results for the introduced tests.
• Local smoke-testing was used during development (manual runs of the demo/server locally), but these are not automated test results.

---

Codex Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fca1d2f61e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Resolve dataset_path before writing template config

The dataset download tool returns local_path as a repo‑relative path (e.g., data/finetune/datasets/...), and _prepare_template writes that value directly into the template config via the {{DATASET_PATH}} replacement. When a real job runs, _run_job executes train.py with cwd=run_dir, so that relative path gets resolved under the run directory (e.g., data/finetune/runs/<id>/data/finetune/datasets/...) and datasets.load_dataset will fail with “file not found” unless the caller provided an absolute path. To avoid real finetune runs breaking on the normal tool flow, resolve to an absolute path or rewrite the dataset path relative to run_dir before templating.

Useful? React with 👍 / 👎.