Synthetic readers for lexical context (addresses #497)

[datvo06]

Closes #497.

Templates expose synthetic read-only tools for lexical symbols that are not already real tools. The LLM can call them to inspect lexical state on demand, instead of receiving the entire scope dumped into the system prompt. Picks up where #545 left off; #585 landed the system-prompt half partially.

Example. A Template defined with a module-global the LLM should be able to see:

from effectful.handlers.llm import Template, LiteLLMProvider
from effectful.ops.semantics import handler
from effectful.ops.types import NotHandled

_known_data = [10, 20, 30, 40, 50]

@Template.define
def report_sum() -> int:
    """Use the `_known_data` tool to read the list of numbers,
    then return their sum as an integer."""
    raise NotHandled

with handler(LiteLLMProvider()):
    result = report_sum()
    assert result == 150

The LLM sees a tool named _known_data whose description is "Read the value of lexical variable _known_data (type list[int]).". It calls the tool, receives the list, and returns the sum.

Dispatch. Used functools.singledispatch to define different readers:

Definition-readers fire for classes and functions. The reader takes level: Literal["short", "full"] and returns str. Short uses pydoc.render_doc (byte-equivalent to help(obj)); full uses inspect.getsource. The probe is inspect.getsource reachability; symbols whose source is unreachable (builtin C, REPL lambdas) are skipped.

Value-readers fire for everything else (the default branch). The reader is zero-arg and returns env[name] live. The probe is pydantic.TypeAdapter(Encodable[T]).json_schema(); any failure causes the symbol to be skipped silently. Catch is broad on purpose, because the probe chains through several third-party libraries (nested_type, inspect.signature, typing.get_overloads, Pydantic schema generation) any of which can crash on third-party objects.

Tool, Agent, and ModuleType values are registered to return None. Tool and Agent are already collected by the existing real-tool path; modules are too big to expose by default.

System prompt. A short static sentence is appended to Template.__system_prompt__ so the LLM knows the read-only-readers category exists. The structured tools array carries per-tool semantics; the preface doesn't enumerate them.

Tests. Invariants pinned by the unit tests:

• A value-reader returns env[name] evaluated at call time, so mutations and rebinds are visible.
• Deleting env[name] after collection causes the reader to raise KeyError on invocation.
• The probe accepts a symbol iff it can be Pydantic-encoded (no false positives, no false negatives across the documented categories).
• Real Tools take precedence over same-named synthetic readers.
• Reader annotations carry no free TypeVars, so the polymorphic-Template substitution machinery from Polymorphic Tool and Template signatures #668 is a no-op for them.
• Definition-reader short form contains the class/function docstring and method signatures; full form contains method bodies (proving it routes to inspect.getsource rather than pydoc).
• A class rebound in env after reader construction is reflected in subsequent definition-reader calls.
• Pydantic BaseModel subclasses route correctly through singledispatch despite having a non-type metaclass.
• Classes / functions whose source is unreachable are skipped at probe time, not at call time.
• TypeVars, modules, and Box values are filtered through their respective skip paths.
• Template.__system_prompt__ contains the preface unconditionally.
• A plain value in the Template's lexical scope is callable as a synthetic reader through Template.tools.
• Two existing assertions in test_template_method and test_template_method_nested_class flip from local_variable not in tools to in, matching the new behavior.

Invariant pinned by the integration test: a Template defined alongside a module-global value, prompted to call the synthetic reader and report a derived value, produces the correct answer end-to-end through a real LLM. The fixture was recorded against gpt-4o-mini and replays cleanly.

[datvo06]

*Filtering, currently, we don't do any filtering, so for example, agent could fetch the API keys through env var if it's in the same lexical context.

API_KEY = ...
@Template.define
def generate_some_thing(...) -> str:
   raise NotHandled

Other than that, we are also including:

• ABCs with pure-Python source (collections.abc.Mapping, Iterable, Callable), Stdlib helpers with reachable source (os.path.join, pathlib.Path) become definition-readers.
• Pydantic-serializable user values, including re.Pattern and pathlib.PosixPath instances, become value-readers.
• Stdlib builtin-C types (int, list, len) fail inspect.getsource and get skipped.
• TypeVars, Term, Operation, Box values fail the Encodable probe and get skipped.

Thinking of adding hide=[...] or expose=[...] as a monkey patch if needed but effect typing #448 will be a cleaner solution.

[eb8680]

Thanks for taking a pass at this. I think it can be quite a bit simpler if you defer most behavior to Encodable. That even includes types like type and types.ModuleType that are currently missing Encodable implementations - they should still trigger Pydantic schema generation errors, which you can catch and use to skip tool generation.

[eb8680]

Also, I don't think this closes #497 in its current form. I don't see any tests for context-sensitivity during synthesis and there's currently nothing to indicate to the LLM that these are lexical variables that are available in generated code. We could leave that behavior for a followup PR to keep this one tractable, although I don't think including it would require much more library code, just more testing.

[eb8680]

This is probably also going to cause problems with notebooks because of all the stuff they inject into globals (cf #578) - maybe we should gate the behavior behind a handler, or just disable it for now when we detect that a Template.__context__ came from a notebook?

[datvo06]

Update: dropped the per-type singledispatch and the definition-reader path. Now one free function _define_lexical_reader builds the tool and _collect_tools catches probe failures inline. In detail:

• Annotated callables flow through as Callable-synthesis tools via _pydantic_callable. Classes, methods, builtins do the same. No per-type code path.
• Filter is name.isidentifier() only - module dunders flow through if encodable.
• Catching Pydantic schema errors so that unencodable things get passed. One caveat is that I had to include these extra errors TypeError/AttributeError/NameError (See nested_type Callable-branch is brittle on values lacking __qualname__/__module__ #673).

One synthesis-oriented test is masked skipping now pending #674. That's likely what happened in RoboTL's https://github.com/BasisResearch/RoboTL/pull/395 and https://github.com/BasisResearch/RoboTL/pull/444 that made mypy type check crash. Will unskip that after #675 and will also fix downstream.

[eb8680]

Thanks for simplifying. I took a closer look at the new tests this time and most of them don't seem very useful - they check implementation details that are likely to change or behavior of upstream features that are not part of this new functionality. I also flagged a design question.

[datvo06]

Update:

1. Make _LexicalVariableTool a subtype of Tool.
2. Removed the mentioned weak tests and strengthened some to identity rather than equality.

Working on gating LexicalVariableTool behind a handler next as _LexicalVariableTool is now a subtype.

[datvo06]

Added:

@Operation.define
def expose_lexical_readers() -> bool:
    """Effect controlling whether `_collect_tools` builds synthetic
    read-only Tools for non-Tool/Template values in a Template's
    lexical scope.

    Default behaviour is *off*: only real Tools/Templates/Agents reach
    the LLM, and the lexical context is invisible.  Install
    `LexicalReaders` to flip it on for the call-site where the LLM
    should be able to inspect closure state.
    """
    return False

This is determining whether the lexical reader tools are being collected.

    with handler(LexicalReaders()):
        assert name in _collect_tools(env)