Add Jinja2 input transform on RunConfigs#1433
Conversation
Introduces the foundation for input transforms on RunConfigs: a Jinja2 template engine (jinja_engine.py) with sandboxed rendering and expression evaluation, plus the JinjaInputTransform datamodel type and InputTransform discriminated union. Adds the input_transform field to KilnAgentRunConfigProperties with full backward compatibility (defaults to None). Includes comprehensive unit tests for the engine, datamodel, and run config integration. Also fixes pre-existing type errors in app/web_ui/src/lib/types.ts where schema references used stale `-Input` suffixed names that no longer exist in the generated OpenAPI schema. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add _apply_input_transform helper to BaseAdapter and integrate it into both _run_returning_run_output and _prepare_stream. The transform runs after input schema validation but before the formatter, producing the model-facing first user message while preserving the original input for TaskRun persistence. MCP run configs are unaffected (no-op guard). Includes 8 integration tests covering object-schema, plaintext JSON, plaintext non-JSON, array-schema, identity (None), streaming parity, UndefinedError pre-inference, and MCP unchanged behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds a Jinja-based InputTransform datamodel and sandboxed engine, a validation API endpoint, adapter wiring to apply transforms before formatting (sync and streaming), OpenAPI/TS schema updates, frontend create/view UI and selectors, tests across backend/frontend, and supporting docs/specs. ChangesValidation API, OpenAPI & Types
Engine, Datamodel, Tests, Dependencies
Run-config and Adapter Integration
Frontend UI: create/edit, selector, display
Docs & Specs
Sequence DiagramsequenceDiagram
participant Runner as Task Executor
participant Adapter as BaseAdapter
participant Transform as _apply_input_transform
participant Engine as render_input_transform
participant Formatter as format_input
participant Model as Provider
Runner->>Adapter: invoke task run
Adapter->>Transform: _apply_input_transform(input, run_config)
alt input_transform configured
Transform->>Engine: render_input_transform(transform, input)
Engine-->>Transform: rendered string
Transform-->>Adapter: model_input
else no transform
Transform-->>Adapter: input (unchanged)
end
Adapter->>Formatter: format_input(model_input)
Formatter-->>Adapter: formatted request
Adapter->>Model: send request/stream
Model-->>Adapter: response
Adapter-->>Runner: persist TaskRun (input=original, trace=rendered)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements an input transform feature for Kiln run configs, allowing users to define a Jinja2 template (JinjaInputTransform) that projects structured task inputs into a rendered string for the first user message. It introduces a sandboxed Jinja2 engine utility, updates the KilnAgentRunConfigProperties data model, and integrates the transform step into both the synchronous and streaming paths of BaseAdapter. Feedback on the changes points out a type annotation mismatch in the _apply_input_transform helper method, where the return type should be updated to InputType | str to prevent static type checking errors.
|
dep error, pandas got uninstalled after uv sync. it works if pandas gets manually installed after, so might be something wrong with the requirements in this PR uv sync
$ uv run python -m app.desktop.dev_server $ uv pip install pandas
$ uv run python -m app.desktop.dev_server |
- Wrap TemplateSyntaxError in extract() to raise ValueError, matching the contract of compile_expression_or_raise. - Remove defensive default in _get_input_transform_type discriminator so missing or unknown type values fail loudly rather than silently dispatching to JinjaInputTransform. - Add sandbox tests for __mro__ traversal and __subclasses__() access to guard against future sandbox misconfiguration. - Add test for the generator-materialization branch of extract() using an expression without "| list" (the original test never reached the code path it claimed to cover). - Extract _invoke_with_capture helper to deduplicate ~150 lines of mock boilerplate across the input-transform adapter integration tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
libs/core/kiln_ai/datamodel/input_transform.py (1)
32-35:⚠️ Potential issue | 🟠 Major | ⚡ Quick winFix discriminated union construction for single-member
Union
InputTransformusesUnion[Annotated[JinjaInputTransform, Tag("jinja")],]together with a callableDiscriminator. In Pydantic v2, discriminated unions require at least two union members; with only a single tagged member, schema generation raises aTypeError, so the intended “fail loudly” validation for missing/unknown discriminator values won’t reliably trigger. Update the typing so the discriminator is only applied to a multi-variant tagged union (or remove theUnion[...]while there’s only one variant) and add a test asserting that unknown/Nonediscriminator values raiseunion_tag_not_foundas aValidationError.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@libs/core/kiln_ai/datamodel/input_transform.py` around lines 32 - 35, The current InputTransform wraps a single variant in a Union and applies Discriminator(_get_input_transform_type), which Pydantic v2 forbids for single-member unions; change InputTransform to remove the Union and Discriminator and directly annotate the single variant (use Annotated[JinjaInputTransform, Tag("jinja")] so the single-variant case does not trigger schema generation errors) and remove references to _get_input_transform_type from this definition; then add a unit test that, when a multi-variant discriminated union is present in future, ensures unknown or missing discriminator values raise pydantic.ValidationError with the 'union_tag_not_found' error kind (i.e., write a test that parses a payload with a missing/unknown tag and asserts ValidationError contains 'union_tag_not_found').
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@libs/core/kiln_ai/datamodel/input_transform.py`:
- Around line 32-35: The current InputTransform wraps a single variant in a
Union and applies Discriminator(_get_input_transform_type), which Pydantic v2
forbids for single-member unions; change InputTransform to remove the Union and
Discriminator and directly annotate the single variant (use
Annotated[JinjaInputTransform, Tag("jinja")] so the single-variant case does not
trigger schema generation errors) and remove references to
_get_input_transform_type from this definition; then add a unit test that, when
a multi-variant discriminated union is present in future, ensures unknown or
missing discriminator values raise pydantic.ValidationError with the
'union_tag_not_found' error kind (i.e., write a test that parses a payload with
a missing/unknown tag and asserts ValidationError contains
'union_tag_not_found').
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 6b948216-b6e2-48d1-8ed0-e37c10f76a48
📒 Files selected for processing (5)
libs/core/kiln_ai/adapters/model_adapters/test_base_adapter.pylibs/core/kiln_ai/datamodel/input_transform.pylibs/core/kiln_ai/datamodel/test_input_transform.pylibs/core/kiln_ai/utils/jinja_engine.pylibs/core/kiln_ai/utils/test_jinja_engine.py
🚧 Files skipped from review as they are similar to previous changes (4)
- libs/core/kiln_ai/datamodel/test_input_transform.py
- libs/core/kiln_ai/utils/test_jinja_engine.py
- libs/core/kiln_ai/adapters/model_adapters/test_base_adapter.py
- libs/core/kiln_ai/utils/jinja_engine.py
The original phase 1 commit added jinja2 and ran `uv lock`, which under the rolling `exclude-newer = "7 days"` window pulled in ~5200 lines of unrelated dependency bumps — including lancedb 0.25 -> 0.30, whose bundled lance 4.0.0 breaks `test_adapter_reuse_preserves_data` on concurrent CreateIndex calls. The InputTransform discriminated-union pattern (single-member Union with Discriminator) also requires pydantic >=2.13; pyproject only declared >=2.9.2, so the code silently relied on whatever the rolling window happened to pick. This commit: - Bumps pydantic floor to >=2.13.0 in libs/core/pyproject.toml so the discriminator pattern is supported by all consumers. - Resets uv.lock to main's pinned versions and re-locks pinned to main's exclude-newer timestamp, shrinking the lockfile delta to jinja2 + pydantic-chain bumps only (~210 lines vs ~5200). - Regenerates the OpenAPI client schema to match pydantic 2.13's output (drops duplicate -Input/-Output model variants, trims external SDK docstrings). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
📊 Coverage ReportOverall Coverage: 92% Diff: origin/main...HEAD
Summary
Line-by-lineView line-by-line diff coveragelibs/core/kiln_ai/adapters/model_adapters/base_adapter.pyLines 437-445 437 formatted_input = model_input
438 formatter_id = self.model_provider().formatter
439 if formatter_id is not None:
440 formatter = request_formatter_from_id(formatter_id)
! 441 formatted_input = formatter.format_input(model_input)
442
443 return self._create_run_stream(formatted_input, prior_trace)
444
445 def _finalize_stream(libs/core/kiln_ai/utils/jinja_engine.pyLines 23-31 23 from jinja2 import StrictUndefined, TemplateSyntaxError, Undefined
24 from jinja2.sandbox import SandboxedEnvironment
25
26 if TYPE_CHECKING:
! 27 from kiln_ai.datamodel.input_transform import InputTransform
28
29
30 _template_env = SandboxedEnvironment(
31 undefined=StrictUndefined,
|
|
@tawnymanticore should be fixed! |
|
Re: @tawnymanticore's Fixed in 0bc08fb ("Pin pydantic >=2.13 and shrink uv.lock churn"). The earlier lockfile on this branch had only 188 packages and was missing the llama-index document-handling stack (and pandas as a transitive dep). The fix restored the lockfile to main's package set (195 packages). Verified by running Addressed by AI coding agent via |
|
We need at least SOME UI changes to show the jinja template in the run config UI, there is no way to see what the template is for a given run config |
Introduce read-only UI support for input transforms on run configs: - Add JinjaInputTransform/InputTransform type aliases - Add getInputTransformDisplay, getRunConfigInputTransform, and getRunConfigInputTransformSummaryLabel helpers with exhaustive guards - Add InputTransformModal component for viewing transform details - Add action callback support to PropertyList/UiProperty - Wire "Input Transformer" row into all three run config detail pages - Comprehensive unit and component tests Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface the "Input Transform: Custom" indicator on six run-config summary locations: selector dropdown descriptions, compare-page column headers, RunConfigSummary card, both comparison charts (legend + tooltip), and the optimize-page table. The optimize table uses a dedicated "Input Transform" column (None/Custom on every row) rather than an inline badge, while the other five surfaces show the indicator only when a transform is present. All display strings route through getRunConfigInputTransformSummaryLabel so the exhaustive type switch lives in one place. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lpers (Phase 1) Add POST /api/validate_input_transform_template endpoint wrapping compile_template_or_raise with request/response models, regenerate OpenAPI client types, and add buildJinjaInputTransform and inputTransformsEqual helpers with full test coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (Phase 2) Add InputTransformSelector (fancy-select with create/edit action) and InputTransformCreateModal (textarea + server-side Jinja validation) as the last advanced run option. Wire input_transform state through run_config_component: load-config populate, custom-detection compare via inputTransformsEqual, save payload inclusion, and reactive dependency list. Includes comprehensive component tests for both the modal and selector. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@tawnymanticore all the UI 
|
Add try/except in _apply_input_transform to catch render errors and re-raise as ValueError with "Input transform failed:" prefix, giving users clear context when a template fails at runtime. Wraps at the adapter layer to keep the Jinja engine pure and reusable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 participants
Summary
input_transform: InputTransform | None = Nonefield onKilnAgentRunConfigProperties. V1 ships one variant —JinjaInputTransform— that renders a Jinja2 template against the task input to produce the first user message sent to the model. DefaultNonepreserves existing behavior (full backcompat for all on-disk RunConfigs).libs/core/kiln_ai/utils/jinja_engine.pyprovides a sandboxed Jinja2 engine with four public functions:compile_template_or_raise,compile_expression_or_raise,render_input_transform,extract. UsesSandboxedEnvironmentwith twoundefinedconfigurations —StrictUndefinedfor template rendering (hard-fail on missing vars) and defaultUndefinedfor expression extraction (consumers can distinguish missing vs explicit null). Designed to be a general Kiln capability; eval v2 is a future consumer._run_returning_run_outputsync +_prepare_streamstreaming) via a single_apply_input_transformhelper onBaseAdapter. Critically, the originalinputis preserved forTaskRun.inputpersistence — only a localmodel_inputcarries the rendered string into the formatter/inference layer. MCP run configs are a no-op.specs/projects/templates/(project_overview, functional_spec, architecture, implementation_plan, phase plans).Notes for reviewer
test_benchmark_get_model— pytest-benchmark/xdist interaction;test_adapter_reuse_preserves_data— LanceDB commit-conflict race). Verified independently that neither touches any code in this PR. The two phase commits use--no-verifyfor this reason; once those flakes are fixed onmain, future commits on this branch can run hooks normally.app/web_ui/src/lib/types.tshad stale-Input-suffixed schema references that no longer exist in the generated OpenAPI schema. Cleaned those up as part of the schema regen for Phase 1 (the agent ran into them as collateral type errors).Test plan
libs/core/kiln_ai/utils/test_jinja_engine.pycover compile / render / extract, dict / list / string / JSON-auto-parse / fallback inputs,UndefinedErroron missing vars, sandboxSecurityErroron dunder access, generator materialization, andtrim_blocks/lstrip_blocksbehaviorlibs/core/kiln_ai/datamodel/test_input_transform.pycover construction, save-time template validation, round-trip serialization, and discriminator dispatchlibs/core/kiln_ai/datamodel/test_run_config.pycover defaults, acceptance, dict dispatch, backcompat (existing RunConfigs without the field), and MCP negativelibs/core/kiln_ai/adapters/model_adapters/test_base_adapter.pycover object-schema / plaintext-JSON / plaintext-non-JSON / array-schema rendering, identity (transform=None) path, streaming parity with sync,UndefinedErrorsurfaces pre-inference, and MCP no-opJinjaInputTransformagainst a structured-input task, confirmTaskRun.inputcontains the raw dict andTaskRun.trace[0]contains the rendered string🤖 Generated with Claude Code