Context
Follow-up to #47. During a real-world plugin authoring session, an LLM made additional mistakes beyond those captured in #47. This issue covers the remaining problems and proposed fixes.
Problem 1: Wildcard assertions defeat forced assertion contract
bigfoot's assertable_fields forces callers to include all required fields in assert_interaction(). But nothing prevents callers from wildcarding every field:
# Satisfies the contract, verifies NOTHING
claude_sdk.assert_query(
prompt=AnyThing(),
model=AnyThing(),
effort=AnyThing(),
result=AnyThing(),
usage=AnyThing(),
is_error=AnyThing(),
)
This is assert True with extra steps. It circumvents the certainty that bigfoot provides. The LLM did this across 27 test assertions to satisfy MissingAssertionFieldsError without computing real expected values.
Proposed fix: Runtime detection
In assert_interaction(), after collecting the expected fields, check if ALL values are IsAnyThing (or duck-type check for always-true matchers):
from dirty_equals import AnyThing
def assert_interaction(self, sentinel, **expected):
# Existing assertable_fields check...
# NEW: detect all-wildcard assertions
if expected and all(isinstance(v, AnyThing) for v in expected.values()):
raise ValueError(
"All assertion fields are wildcards (AnyThing). "
"This assertion verifies nothing. Use real expected values.\n\n"
"If you need to wildcard SOME fields, that's fine:\n"
f" {sentinel.source_id}.assert_*(field1='real', field2=AnyThing())\n\n"
"But wildcarding ALL fields defeats bigfoot's purpose."
)
This allows partial wildcards (useful for single-responsibility tests) while blocking the degenerate case.
Proposed docs addition
In the writing-plugins guide, under "Convenience assertion methods":
Anti-pattern: wildcard assertions. Never use AnyThing() or equivalent for ALL fields. Wildcarding some fields for single-responsibility tests is fine. Wildcarding all fields verifies nothing.
# WRONG - verifies nothing
plugin.assert_query(prompt=AnyThing(), model=AnyThing(), result=AnyThing())
# OK - verifies prompt, wildcards the rest for a focused test
plugin.assert_query(prompt="Review this PR", model=AnyThing(), result=AnyThing())
# BEST - verifies everything
plugin.assert_query(prompt="Review this PR", model="claude-opus-4-6", result="Done.")
Also add to module-level docstring (proposed in #47):
Anti-patterns:
...existing items...
- NEVER wildcard ALL fields in assert_* calls. Partial wildcards OK,
all-wildcard verifies nothing.
Problem 2: Docs cross-linking gaps
The LLM read the writing-plugins guide but missed critical information in the pytest-integration and configuration guides. The guides don't cross-reference each other effectively.
Specific gaps found
-
writing-plugins.md shows verifier.sandbox() in the "manual use outside pytest" section. The LLM grabbed this pattern for pytest tests. The guide should prominently link to pytest-integration.md at the top:
Using pytest? See pytest integration for the standard with bigfoot: pattern. The manual StrictVerifier() pattern below is for use outside pytest only.
-
writing-plugins.md doesn't mention how to disable built-in plugins that conflict with custom plugins. The LLM created a bare verifier to avoid built-in socket/subprocess plugins instead of configuring them away. Add a section:
Disabling built-in plugins: If built-in plugins interfere with your custom plugin's tests, disable them in pyproject.toml rather than creating custom verifiers:
[tool.bigfoot]
disabled_plugins = ["socket", "subprocess"]
-
configuration.md doesn't document enabled_plugins or disabled_plugins options. If these exist, they should be documented. If they don't exist, they should be added (see below).
-
README on PyPI should link to the key guides with one-line descriptions so LLMs doing web searches find the right docs:
## Documentation
- [Quick Start](docs/guides/quickstart.md)
- [pytest Integration](docs/guides/pytest-integration.md) - `with bigfoot:`, autouse fixtures
- [Writing Plugins](docs/guides/writing-plugins.md) - Custom plugins for any library
- [Configuration](docs/guides/configuration.md) - Plugin selection, guard mode
- [Guard Mode](docs/guides/guard-mode.md) - Block unmocked I/O
Problem 3: bigfoot_verifier fixture used instead of bigfoot.current_verifier()
The pytest-integration guide labels bigfoot_verifier as an "escape hatch" (line 76), but the writing-plugins guide shows it in the custom plugin fixture pattern. The LLM used bigfoot_verifier in fixtures for custom plugins because it appeared in the first code example they found.
Proposed fix
In writing-plugins.md, the "Registering and using the plugin" section already shows bigfoot.current_verifier() correctly. But add a callout:
Do not use bigfoot_verifier fixture in your plugin fixtures. Use bigfoot.current_verifier() instead. The bigfoot_verifier fixture is an escape hatch for direct verifier access, not the standard plugin registration pattern.
Problem 4: Pyright can't resolve bigfoot's dynamic API
bigfoot uses __getattr__ on the module for with bigfoot:, bigfoot.current_verifier(), bigfoot.http, etc. Pyright reports these as errors:
Object of type "Module("bigfoot")" cannot be used with "with"
"current_verifier" is not a known attribute of module "bigfoot"
"StrictVerifier" is unknown import symbol
These false positives push LLMs toward using private imports (which Pyright CAN resolve) instead of the public API.
Proposed fix
Add a py.typed marker and __init__.pyi stub file that declares the public API:
# bigfoot/__init__.pyi
from bigfoot._verifier import StrictVerifier as StrictVerifier
from bigfoot._errors import UnmockedInteractionError as UnmockedInteractionError
from bigfoot._errors import UnassertedInteractionsError as UnassertedInteractionsError
# ...etc
def current_verifier() -> StrictVerifier: ...
def sandbox() -> SandboxContext: ...
def __enter__() -> StrictVerifier: ...
def __exit__(exc_type, exc_val, exc_tb) -> None: ...
This is also tracked in #32 (stub files) but worth reiterating here since it directly caused anti-patterns.
Context
Follow-up to #47. During a real-world plugin authoring session, an LLM made additional mistakes beyond those captured in #47. This issue covers the remaining problems and proposed fixes.
Problem 1: Wildcard assertions defeat forced assertion contract
bigfoot's
assertable_fieldsforces callers to include all required fields inassert_interaction(). But nothing prevents callers from wildcarding every field:This is
assert Truewith extra steps. It circumvents the certainty that bigfoot provides. The LLM did this across 27 test assertions to satisfyMissingAssertionFieldsErrorwithout computing real expected values.Proposed fix: Runtime detection
In
assert_interaction(), after collecting the expected fields, check if ALL values areIsAnyThing(or duck-type check for always-true matchers):This allows partial wildcards (useful for single-responsibility tests) while blocking the degenerate case.
Proposed docs addition
In the writing-plugins guide, under "Convenience assertion methods":
Also add to module-level docstring (proposed in #47):
Problem 2: Docs cross-linking gaps
The LLM read the writing-plugins guide but missed critical information in the pytest-integration and configuration guides. The guides don't cross-reference each other effectively.
Specific gaps found
writing-plugins.md shows
verifier.sandbox()in the "manual use outside pytest" section. The LLM grabbed this pattern for pytest tests. The guide should prominently link to pytest-integration.md at the top:writing-plugins.md doesn't mention how to disable built-in plugins that conflict with custom plugins. The LLM created a bare verifier to avoid built-in socket/subprocess plugins instead of configuring them away. Add a section:
configuration.md doesn't document
enabled_pluginsordisabled_pluginsoptions. If these exist, they should be documented. If they don't exist, they should be added (see below).README on PyPI should link to the key guides with one-line descriptions so LLMs doing web searches find the right docs:
Problem 3:
bigfoot_verifierfixture used instead ofbigfoot.current_verifier()The pytest-integration guide labels
bigfoot_verifieras an "escape hatch" (line 76), but the writing-plugins guide shows it in the custom plugin fixture pattern. The LLM usedbigfoot_verifierin fixtures for custom plugins because it appeared in the first code example they found.Proposed fix
In writing-plugins.md, the "Registering and using the plugin" section already shows
bigfoot.current_verifier()correctly. But add a callout:Problem 4: Pyright can't resolve bigfoot's dynamic API
bigfoot uses
__getattr__on the module forwith bigfoot:,bigfoot.current_verifier(),bigfoot.http, etc. Pyright reports these as errors:These false positives push LLMs toward using private imports (which Pyright CAN resolve) instead of the public API.
Proposed fix
Add a
py.typedmarker and__init__.pyistub file that declares the public API:This is also tracked in #32 (stub files) but worth reiterating here since it directly caused anti-patterns.