Skip to content
This repository was archived by the owner on Feb 11, 2026. It is now read-only.
This repository was archived by the owner on Feb 11, 2026. It is now read-only.

Add integration tests for all documented user examples #215

@turbomam

Description

@turbomam

Problem

During documentation audit, we discovered several documented CLI commands that don't actually exist or work. We need automated testing to prevent this.

What Happened

Documentation claimed these commands existed:

  • get-paper-citations - doesn't exist
  • get-paper-references - doesn't exist
  • get-citation-network - doesn't exist
  • get-all-identifiers - wrong name (should be get-all-identifiers-from-europepmc)

These were documented for months without anyone noticing.

Root Cause

  • No automated testing of documentation examples
  • CLI and MCP tools aren't in sync
  • No CI/CD validation that documented commands actually work

Proposed Solution

1. Create Documentation Test Suite

Add tests/test_documentation_examples.py:

"""Test that all examples in README.md and USERS.md actually work."""

def test_readme_cli_examples():
    """Test every CLI command shown in README.md."""
    # Extract code blocks from README
    # Run each command
    # Verify it doesn't error
    
def test_users_md_cli_examples():
    """Test every CLI command shown in USERS.md."""
    
def test_mcp_tools_listed_in_readme():
    """Verify MCP tool list matches actual registered tools."""
    # Parse list from README
    # Get actual registered tools from main.py
    # Assert they match

2. Add CI Check

Add to .github/workflows/:

- name: Test documentation examples
  run: |
    pytest tests/test_documentation_examples.py

3. Add Pre-commit Hook

Optionally add hook that:

  • Extracts CLI commands from markdown
  • Verifies they exist in artl-cli --help
  • Fails if mismatch found

4. Create Command Inventory

Maintain commands.json:

{
  "mcp_tools": ["search_europepmc_papers", ...],
  "cli_commands": ["get-doi-metadata", ...],
  "both": ["search-papers-by-keyword", ...]
}

Update this automatically and verify docs match.

Testing Levels

  1. Smoke tests - Does command exist? Does it return non-error?
  2. Integration tests - Does it work with real API calls?
  3. Example validation - Does the exact example in docs work?

Benefits

  • Prevent documenting non-existent features
  • Catch CLI/MCP sync issues early
  • Ensure examples stay up-to-date
  • Build confidence in documentation accuracy

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions