Skip to content

Prevent wildcard assertion anti-pattern and improve cross-linking in docs #48

@elijahr

Description

@elijahr

Context

Follow-up to #47. During a real-world plugin authoring session, an LLM made additional mistakes beyond those captured in #47. This issue covers the remaining problems and proposed fixes.

Problem 1: Wildcard assertions defeat forced assertion contract

bigfoot's assertable_fields forces callers to include all required fields in assert_interaction(). But nothing prevents callers from wildcarding every field:

# Satisfies the contract, verifies NOTHING
claude_sdk.assert_query(
    prompt=AnyThing(),
    model=AnyThing(),
    effort=AnyThing(),
    result=AnyThing(),
    usage=AnyThing(),
    is_error=AnyThing(),
)

This is assert True with extra steps. It circumvents the certainty that bigfoot provides. The LLM did this across 27 test assertions to satisfy MissingAssertionFieldsError without computing real expected values.

Proposed fix: Runtime detection

In assert_interaction(), after collecting the expected fields, check if ALL values are IsAnyThing (or duck-type check for always-true matchers):

from dirty_equals import AnyThing

def assert_interaction(self, sentinel, **expected):
    # Existing assertable_fields check...
    
    # NEW: detect all-wildcard assertions
    if expected and all(isinstance(v, AnyThing) for v in expected.values()):
        raise ValueError(
            "All assertion fields are wildcards (AnyThing). "
            "This assertion verifies nothing. Use real expected values.\n\n"
            "If you need to wildcard SOME fields, that's fine:\n"
            f"    {sentinel.source_id}.assert_*(field1='real', field2=AnyThing())\n\n"
            "But wildcarding ALL fields defeats bigfoot's purpose."
        )

This allows partial wildcards (useful for single-responsibility tests) while blocking the degenerate case.

Proposed docs addition

In the writing-plugins guide, under "Convenience assertion methods":

Anti-pattern: wildcard assertions. Never use AnyThing() or equivalent for ALL fields. Wildcarding some fields for single-responsibility tests is fine. Wildcarding all fields verifies nothing.

# WRONG - verifies nothing
plugin.assert_query(prompt=AnyThing(), model=AnyThing(), result=AnyThing())

# OK - verifies prompt, wildcards the rest for a focused test
plugin.assert_query(prompt="Review this PR", model=AnyThing(), result=AnyThing())

# BEST - verifies everything
plugin.assert_query(prompt="Review this PR", model="claude-opus-4-6", result="Done.")

Also add to module-level docstring (proposed in #47):

Anti-patterns:
    ...existing items...
    - NEVER wildcard ALL fields in assert_* calls. Partial wildcards OK,
      all-wildcard verifies nothing.

Problem 2: Docs cross-linking gaps

The LLM read the writing-plugins guide but missed critical information in the pytest-integration and configuration guides. The guides don't cross-reference each other effectively.

Specific gaps found

  1. writing-plugins.md shows verifier.sandbox() in the "manual use outside pytest" section. The LLM grabbed this pattern for pytest tests. The guide should prominently link to pytest-integration.md at the top:

    Using pytest? See pytest integration for the standard with bigfoot: pattern. The manual StrictVerifier() pattern below is for use outside pytest only.

  2. writing-plugins.md doesn't mention how to disable built-in plugins that conflict with custom plugins. The LLM created a bare verifier to avoid built-in socket/subprocess plugins instead of configuring them away. Add a section:

    Disabling built-in plugins: If built-in plugins interfere with your custom plugin's tests, disable them in pyproject.toml rather than creating custom verifiers:

    [tool.bigfoot]
    disabled_plugins = ["socket", "subprocess"]
  3. configuration.md doesn't document enabled_plugins or disabled_plugins options. If these exist, they should be documented. If they don't exist, they should be added (see below).

  4. README on PyPI should link to the key guides with one-line descriptions so LLMs doing web searches find the right docs:

    ## Documentation
    - [Quick Start](docs/guides/quickstart.md)
    - [pytest Integration](docs/guides/pytest-integration.md) - `with bigfoot:`, autouse fixtures
    - [Writing Plugins](docs/guides/writing-plugins.md) - Custom plugins for any library  
    - [Configuration](docs/guides/configuration.md) - Plugin selection, guard mode
    - [Guard Mode](docs/guides/guard-mode.md) - Block unmocked I/O

Problem 3: bigfoot_verifier fixture used instead of bigfoot.current_verifier()

The pytest-integration guide labels bigfoot_verifier as an "escape hatch" (line 76), but the writing-plugins guide shows it in the custom plugin fixture pattern. The LLM used bigfoot_verifier in fixtures for custom plugins because it appeared in the first code example they found.

Proposed fix

In writing-plugins.md, the "Registering and using the plugin" section already shows bigfoot.current_verifier() correctly. But add a callout:

Do not use bigfoot_verifier fixture in your plugin fixtures. Use bigfoot.current_verifier() instead. The bigfoot_verifier fixture is an escape hatch for direct verifier access, not the standard plugin registration pattern.

Problem 4: Pyright can't resolve bigfoot's dynamic API

bigfoot uses __getattr__ on the module for with bigfoot:, bigfoot.current_verifier(), bigfoot.http, etc. Pyright reports these as errors:

Object of type "Module("bigfoot")" cannot be used with "with" 
"current_verifier" is not a known attribute of module "bigfoot"
"StrictVerifier" is unknown import symbol

These false positives push LLMs toward using private imports (which Pyright CAN resolve) instead of the public API.

Proposed fix

Add a py.typed marker and __init__.pyi stub file that declares the public API:

# bigfoot/__init__.pyi
from bigfoot._verifier import StrictVerifier as StrictVerifier
from bigfoot._errors import UnmockedInteractionError as UnmockedInteractionError
from bigfoot._errors import UnassertedInteractionsError as UnassertedInteractionsError
# ...etc

def current_verifier() -> StrictVerifier: ...
def sandbox() -> SandboxContext: ...
def __enter__() -> StrictVerifier: ...
def __exit__(exc_type, exc_val, exc_tb) -> None: ...

This is also tracked in #32 (stub files) but worth reiterating here since it directly caused anti-patterns.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions