Parametric LLM model tests (#617)

* Make LLM tests fully parametric in model name via EFFECTFUL_LLM_MODEL env var

Replace provider-specific environment variable checks (OPENAI_API_KEY,
ANTHROPIC_API_KEY) and skip markers (requires_openai, requires_anthropic)
with a single EFFECTFUL_LLM_MODEL environment variable that controls
which model is used for all LLM integration tests.

- Remove hardcoded model names from all test files in favor of LLM_MODEL
  read from EFFECTFUL_LLM_MODEL env var
- Replace requires_openai/requires_anthropic markers with requires_llm
- Remove model parametrization that was cross-provider; tests now use
  whichever model the env var specifies
- Use litellm.supports_vision() to conditionally skip vision tests
- Remove default model from LiteLLMProvider (make model required)
- Update CI workflow to pass EFFECTFUL_LLM_MODEL as a matrix parameter,
  making it easy to add parallel CI stages for different providers
- Rename/remove fixture files to match updated test names

Closes #589

* Deduplicate LLM_MODEL/requires_llm into conftest, fix import order, rename test

- Move LLM_MODEL and requires_llm definitions to tests/conftest.py;
  all four LLM test files now import from there
- Fix import ordering in test_handlers_llm_encoding.py (stdlib before local)
- Rename test_agent_tool_names_are_openai_compatible_integration to
  test_agent_tool_names_are_valid_integration since it's no longer
  OpenAI-specific

* Address review feedback: fix breaking change, import hygiene, redundancy

- Restore LiteLLMProvider default model via env var fallback so existing
  callers (template.py, test_handlers_llm_template.py, notebook) are not
  broken: model=os.environ.get("EFFECTFUL_LLM_MODEL", "gpt-4o")
- Move LLM_MODEL/requires_llm from conftest.py to tests/_llm_helpers.py
  since conftest.py should not be imported directly
- Fix import placement in test_handlers_llm_provider.py (was between
  module-level constants instead of grouped with imports)
- Remove redundant @requires_llm on vision tests where the skipif
  condition already covers the not-LLM_MODEL case

* Revert LiteLLMProvider default to plain literal "gpt-4o"

The env var belongs in test infrastructure, not the library API.
LiteLLMProvider should have a clean, explicit default.

* Minor fix

* minor fixes

* minor

* Make LLM test suite parametric in model name (#589)

Remove provider-specific environment variable checks and skip markers
from LLM tests in favor of a single EFFECTFUL_LLM_MODEL env var.

- Add LLM_MODEL and requires_llm to tests/conftest.py; LLM_MODEL
  defaults to "gpt-4o-mini" and is overridable via EFFECTFUL_LLM_MODEL
- Live tests (tool calling, encoding, agent tool names) use LLM_MODEL
  and are gated by requires_llm which checks for any provider API key
- Integration tests use make_provider() which returns a live
  LiteLLMProvider when API keys are available, else falls back to
  ReplayLiteLLMProvider for offline replay from fixtures
- Replay-only tests (simple_prompt_multiple_models, cross_endpoint,
  caching) keep hardcoded model names and always run since they never
  call the API
- Update CI workflow to pass EFFECTFUL_LLM_MODEL as a matrix parameter
  for easy parallel stages with different providers

Closes #589

* Minor fix

* Minor fix

* Fix tuple type schemas for OpenAI structured output compatibility

Three fixes in encoding.py:

1. TupleEncodable.encode() now returns a TupleItems model instance
   (not a raw tuple), and deserialize() returns the model directly.
   This fixes pydantic validation in litellm integration tests for
   NamedTuple and fixed-tuple types.

2. Add _TupleSafeJsonSchema that overrides pydantic's tuple_schema()
   to produce object schemas (item_0, item_1 properties) instead of
   prefixItems arrays. Applied via _BoxEncoding.model_json_schema()
   so dataclasses containing tuple fields produce OpenAI-compatible
   schemas.

3. SequenceEncodable.encode() returns a list (not tuple) to preserve
   encode idempotency — nested_type on a list dispatches to the
   sequence encoder, avoiding a mismatch with TupleEncodable.

Also adds test_handlers_llm_encoding.py back to CI workflow.

* Clean up

* Split out tuple encoding fixes to separate branch

Revert encoding.py to master and remove encoding tests from CI
workflow. Tuple schema fixes will be in a dedicated PR.

* Removing hard-coded test value and renaminng const

* Use litellm to check requires vision

* Fix missing fixture and lint formatting

- Add renamed fixture for test_simple_prompt (was test_simple_prompt_multiple_models[gpt-4o-mini])
- Reformat long lines to pass ruff lint

Author: datvo06

.github/workflows/test_llm.yml

@@ -14,6 +14,7 @@ jobs:
     strategy:
       matrix:
         python-version: ["'3.13'"]
+        model: ["gpt-4o-mini"]
     steps:
       - uses: actions/checkout@v4
 
@@ -34,7 +35,8 @@ jobs:
 
       - name: Run LLM integration tests
         env:
+          EFFECTFUL_LLM_MODEL: ${{ matrix.model }}
           OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
           ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
         run: |
-          uv run pytest tests/test_handlers_llm_provider.py -v --tb=short
+          uv run pytest tests/test_handlers_llm_provider.py tests/test_handlers_llm_tool_calling_poem.py tests/test_handlers_llm_tool_calling_book.py -v --tb=short

tests/conftest.py

@@ -1,5 +1,24 @@
+import os
+
+import litellm
 import pytest
 
+EFFECTFUL_LLM_MODEL = os.environ.get("EFFECTFUL_LLM_MODEL", "gpt-4o-mini")
+
+_HAS_LLM_API_KEY = litellm.validate_environment(model=EFFECTFUL_LLM_MODEL)[
+    "keys_in_environment"
+]
+
+requires_llm = pytest.mark.skipif(
+    not _HAS_LLM_API_KEY,
+    reason=f"No API key configured for model {EFFECTFUL_LLM_MODEL}",
+)
+
+requires_vision = pytest.mark.skipif(
+    not litellm.supports_vision(model=EFFECTFUL_LLM_MODEL),
+    reason=f"Model {EFFECTFUL_LLM_MODEL} does not support vision",
+)
+
 UNIMPLEMENTED_SUBSTRINGS = [
     "infer.JitTrace_ELBO",
     "the event_dim arg",

…prompt_multiple_models[gpt-4o-mini].json …LiteLLMProvider__test_simple_prompt.jsontests/fixtures/tests_test_handlers_llm_provider.py__TestLiteLLMProvider__test_simple_prompt_multiple_models[gpt-4o-mini].json renamed to tests/fixtures/tests_test_handlers_llm_provider.py__TestLiteLLMProvider__test_simple_prompt.json tests/fixtures/tests_test_handlers_llm_provider.py__TestLiteLLMProvider__test_simple_prompt_multiple_models[gpt-4o-mini].json renamed to tests/fixtures/tests_test_handlers_llm_provider.py__TestLiteLLMProvider__test_simple_prompt.json

@@ -28,9 +28,7 @@
 from effectful.internals.unification import nested_type
 from effectful.ops.semantics import handler
 from effectful.ops.types import Operation, Term
-from tests.test_handlers_llm_tool_calling_book import requires_openai
-
-CHEAP_MODEL = "gpt-4o-mini"
+from tests.conftest import EFFECTFUL_LLM_MODEL, requires_llm
 
 # ---------------------------------------------------------------------------
 # Module-level type definitions
@@ -744,14 +742,14 @@ def _encode_tool_spec(tool: Tool[..., Any]) -> dict[str, Any]:
     raise TypeError(f"Unexpected encoded tool spec type: {type(tool_spec_obj)}")
 
 
-@requires_openai
+@requires_llm
 @pytest.mark.parametrize("ty,_value,ctx", PROVIDER_CASES)
 def test_litellm_completion_accepts_encodable_response_model_for_supported_types(
     ty: Any, _value: Any, ctx: Mapping[str, Any] | None
 ) -> None:
     enc = Encodable.define(ty, ctx)
     kwargs: dict[str, Any] = {
-        "model": CHEAP_MODEL,
+        "model": EFFECTFUL_LLM_MODEL,
         "messages": [
             {
                 "role": "user",
@@ -777,7 +775,7 @@ def test_litellm_completion_accepts_encodable_response_model_for_supported_types
     pydantic.TypeAdapter(enc.base).validate_python(decoded)
 
 
-@requires_openai
+@requires_llm
 @pytest.mark.parametrize("ty,_value,ctx", PROVIDER_CASES)
 def test_litellm_completion_accepts_tool_with_type_as_param(
     ty: Any, _value: Any, ctx: Mapping[str, Any] | None
@@ -793,7 +791,7 @@ def _fn(value):
 
     tool: Tool[..., Any] = Tool.define(_fn)
     response = litellm.completion(
-        model=CHEAP_MODEL,
+        model=EFFECTFUL_LLM_MODEL,
         messages=[{"role": "user", "content": "Return hello, do NOT call any tools."}],
         tools=[_encode_tool_spec(tool)],
         tool_choice="none",
@@ -802,7 +800,7 @@ def _fn(value):
     assert response is not None
 
 
-@requires_openai
+@requires_llm
 @pytest.mark.parametrize("ty,_value,ctx", PROVIDER_CASES)
 def test_litellm_completion_accepts_tool_with_type_as_return(
     ty: Any, _value: Any, ctx: Mapping[str, Any] | None
@@ -818,7 +816,7 @@ def _fn():
 
     tool: Tool[..., Any] = Tool.define(_fn)
     response = litellm.completion(
-        model=CHEAP_MODEL,
+        model=EFFECTFUL_LLM_MODEL,
         messages=[{"role": "user", "content": "Return hello, do NOT call any tools."}],
         tools=[_encode_tool_spec(tool)],
         tool_choice="none",