Unit Test Research

[JasonUranta]

Overview

We need to research and have unit tests for our project, so that we can identify when code has failed

Action Items

• Go to ChatGPT and give it a prompt

  I am going to give you a requirements document for a project I am working on.  We want to add Unit Testing to the project coding.  Please ask me questions, so that you can provide Unit Testing package options.  We are looking for something free.
  

• feed in the requirements document for the ai-skills-assessor repo
• Have a conversation with ChatGPT about the testing library and testing library options
• Document recommendations and unanswered questions in a comment on this issue

Resources

Link to chatGPT conversation:

• https://chatgpt.com/share/68b75888-290c-800e-a897-a2e90185efa4

Requirements Doc link:

• https://github.com/hackforla/ai-skills-assessor/blob/mixin/AISA%20Requirements%20Doc%20v2.docx
• https://github.com/hackforla/ai-skills-assessor/blob/mixin/Ai%20Skills%20Requirements.pdf

[ExperimentsInHonesty]

@JasonUranta I cleaned up the issue. Please go find the requirments doc link and add it to the Resources section of this issue

[JasonUranta]

Instructions

1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
2. Blockers: "Difficulties or errors encountered."
3. Availability: "How much time will you have this week to work on this issue?"
4. ETA: "When do you expect this issue to be completed?"
5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."

1. Progress: Had a conversation with ChatGPT after feeding it the AISA Requirements Doc and POC doc to determine the needed testing framework, libraries, and modules to support Test Driven Development of the AI Skills Assessment Project
2. Blockers: Had less time to work than I thought I would this week, so a bit late with this update. Wasn't sure if I should include the POC doc, but opted to do so so that ChatGPT would have enough contextual information
3. Availability: Should be available over the weekend
4. ETA: N/A
5. Pictures (if necessary):
   No picture, but here is the final response/recommendation from ChatGPT:

Awesome—thanks, this is plenty to give you solid, free options tailored to your Python rewrite and workflow (scripts + optional CLI/UI, unit + integration, TDD, error-logging coverage, and file schema checks).

Recommended (all free) Python testing stack

Core

• pytest – fast, expressive tests, fixtures, great plugin ecosystem.
• coverage.py + pytest-cov – coverage reports (terminal + XML for CI badges/gates).
• tox or nox – run test/lint/typing locally in isolated envs (mirrors CI).
• pre-commit – auto-run formatting/lint hooks before commits to keep tests green.

Mocking & API simulation

• pytest-mock (thin wrapper over unittest.mock) – easy patching/mocking.
• responses or requests-mock – simulate GitHub/OpenAI HTTP calls.
• vcrpy – record/replay HTTP cassettes for stable integration tests without hitting real APIs.

Property-based & edge cases

• hypothesis – automatically generates edge-case inputs (great for parsing, throttling logic, CSV handling).

Logging & error paths

• Built-in caplog fixture (pytest) to assert log level/messages for your “post error code/message” requirements.

CLI/UI testing

• If CLI uses Typer or Click: use Typer/Click’s CliRunner.
  Otherwise, use subprocess.run with temporary dirs/fixtures.

File schema/content validation

• jsonschema – validate JSON outputs written to /Data.
• frictionless (lightweight) – validate CSV schema and constraints.
  (Alternative: Great Expectations is free but heavier; start with frictionless.)

Type & style (keeps tests reliable)

• ruff (lint/format), mypy (static typing), black (if you prefer strict formatting).

---

Why this stack fits your requirements

• Python-first & TDD-friendly: pytest + hypothesis lets you write tests first and iterate quickly.
• Covers unit + integration: mock HTTP for fast unit tests; vcrpy for deterministic “integration-like” tests without secrets/rate limits.
• Validates outputs: jsonschema/frictionless assert your JSON/CSV shapes match the /Data contract in the docs.
• Error-logging coverage: caplog makes it easy to assert your structured logs and “public error posting” pathways are exercised.
• Fast, free, widely adopted: everything here is open source and battle-tested.

---

“Good, Better, Best” package sets

Good (smallest set to start)

• pytest, pytest-cov, pytest-mock, responses, jsonschema, frictionless

Better (adds resilience & TDD power)

• • hypothesis, vcrpy, tox (or nox), pre-commit, ruff, mypy

Best (polish & scale)

• • pytest-xdist (parallel tests), pytest-rerunfailures (flaky net tests), coverage threshold gates in CI

---

Project scaffolding

ai_skills_assessor/
  src/...
  tests/
    unit/
      test_parsing.py
      test_rate_limit.py
      test_logging.py
      test_cli.py
    integration/
      test_github_api_vcr.py       # uses vcrpy cassettes
      test_openai_api_vcr.py
    schemas/
      data_json.schema.json        # for jsonschema
      data_csv.resource.json       # for frictionless Table Schema
  .pre-commit-config.yaml
  pyproject.toml                   # deps + tool configs
  tox.ini (or noxfile.py)

Example snippets (brief):

Assert logging with caplog:

def test_logs_error_and_returns_fallback(caplog, my_module):
    with caplog.at_level("ERROR"):
        result = my_module.process_issue(issue_id="123")  # triggers error path
    assert "GitHub API" in caplog.text

Mock HTTP fast path:

import responses
@responses.activate
def test_fetch_comments_handles_403():
    responses.add(responses.GET, "https://api.github.com/...", status=403, json={"message":"rate limit"})
    assert fetch_comments("...") == []

Validate JSON/CSV outputs:

from jsonschema import validate, Draft202012Validator
def test_json_schema(tmp_path):
    out = generate_output(...)
    validate(out, load_schema("schemas/data_json.schema.json"), cls=Draft202012Validator)

---

CI vs developer-run (your Q6)

Do both.

• Developer-run via tox/nox ensures reproducible local runs that match CI.
• GitHub Actions runs on every PR/push to keep main green and catch regressions early, which supports your “daily automated scripts” posture.

Recommended GitHub Actions workflow (concise):

name: CI
on:
  pull_request:
  push:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix: { python-version: ['3.10','3.11','3.12'] }
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: ${{ matrix.python-version }} }
      - uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-${{ matrix.python-version }}-${{ hashFiles('**/pyproject.toml') }}
      - run: pip install -e .[dev]         # include your test deps
      - run: pytest -q --cov=src --cov-report=xml --cov-fail-under=85

Secrets & integration tests in CI

• Keep unit tests fully mocked, always run.
• Gate live API tests behind an env var (e.g., RUN_LIVE_API=true) and a separate workflow that has access to secrets (PAT, OpenAI key). Default to off so forks/PRs are safe.
• Prefer vcrpy recordings for reproducible “integration” suites without secrets or rate limits.

---

Test plan tied to your requirements

1. Unit tests (fast, mocked)

   • Parsing issue overview/action items; CSV taxonomy parsing.
   • Throttling/rate-limit backoff logic (simulate 403/429).
   • Prompt assembly and token trimming before OpenAI calls.
   • File writers: JSON/CSV emit exact schema to /Data.
   • Error posting payload construction (don’t call GitHub; assert payload).

2. Integration-style (deterministic)

   • vcrpy cassettes for GitHub Issue API fetch/paginate, posting comments.
   • vcrpy/OpenAI cassettes for small prompt/responses (or mock with golden files for cost control).

3. Negative paths & observability

   • Assert logs for GitHub/OpenAI timeouts, 5xx, schema failures, and fallback behavior.

4. Quality gates

   • Coverage ≥ 85% with per-module minimums for critical code (fetch, classify, write).
   • Lint + type check pass required to merge.

---

Minimal pyproject.toml test extras (example)

[project.optional-dependencies]
dev = [
  "pytest",
  "pytest-cov",
  "pytest-mock",
  "responses",
  "vcrpy",
  "hypothesis",
  "jsonschema",
  "frictionless",
  "ruff",
  "mypy",
  "pre-commit",
  "tox"  # or "nox"
]

---

Final picks (TL;DR)

• Primary: pytest, pytest-cov, pytest-mock
• HTTP: responses (unit), vcrpy (integration-like)
• Edge cases: hypothesis
• Schemas: jsonschema (JSON), frictionless (CSV)
• Workflow: tox/nox, pre-commit, ruff, mypy
• CI: GitHub Actions (run unit tests always; optional live tests behind flag)

If you want, I can draft:

• a ready-to-drop tests/ skeleton with first 3 tests,
• JSON/CSV schema stubs for /Data, and
• a complete CI YAML + tox.ini to match.

No extra info needed on my end to proceed, but if you already know which CLI library you’re using (Typer/Click/argparse), I’ll tailor the CLI test helpers accordingly.

[Jason-Uranta]

Overview

We need to research and have unit tests for our project, so that we can identify when code has failed

Action Items

• Go to ChatGPT and give it a prompt

  I am going to give you a requirements document for a project I am working on.  We want to add Unit Testing to the project coding.  Please ask me questions, so that you can provide Unit Testing package options.  We are looking for something free.
  

  [x] feed in the requirements document for the ai-skills-assessor repo[x] Have a conversation with ChatGPT about the testing library and testing library options[x] Document recommendations and unanswered questions in a comment on this issue

Resources

Link to chatGPT conversation:

• https://chatgpt.com/share/68b75888-290c-800e-a897-a2e90185efa4

Requirements Doc link:

• https://github.com/hackforla/ai-skills-assessor/blob/mixin/AISA%20Requirements%20Doc%20v2.docx
• https://github.com/hackforla/ai-skills-assessor/blob/mixin/Ai%20Skills%20Requirements.pdf

Here is an updated link to the ChatGPT conversation where I feed Chat the YAML and Python files used in Issue #4 to give more context for test suite:
https://chatgpt.com/share/68c8cb5f-94c8-800e-b581-cb7eaa0223ad

Here is the most recent response:

Short answer: your stack recommendation mostly stands (pytest + pytest-cov + pytest-mock + responses + vcrpy + hypothesis + jsonschema + frictionless + ruff + mypy + tox/nox + pre-commit), but I’d tweak how we wire tests based on your code’s structure and your GitHub Actions YAML.

Here’s what I’d change or add.

What to tweak based on your code

1. Avoid import-time side effects (critical for testing)

• Right now you read GITHUB_TOKEN/GH_TOKEN and can SystemExit at import time. That makes unit tests brittle.
• Refactor: move token/env resolution into main() (or a get_config() function). Keep module import side-effect-free.
• Same for computing owner/repo from env—defer until runtime.

2. Carve out pure helpers for cheap, fast unit tests

• The following are perfect to test in isolation (no I/O):
  _split_owner_repo, _parse_iso, _iso, _min_updated, decrement_one_second, _parse_retry_after, trim_user, trim_comments, to_assessment_record, to_assessment_record_from_review, last_seen_by_endpoint, next_url, _users_signature, _config_signature.
• Put them in scripts/fetch_utils.py or keep them here but ensure import has no env requirements.

3. Filesystem-touching bits should take paths as params

• You already do this for many helpers—good! Keep OUTPUT_PATH injectable. Tests will use tmp_path and pass paths explicitly to load_existing, save_json_atomic, ETag/meta helpers.

4. HTTP layer testability

• build_session sets retries and headers—unit test with pytest-mock to assert headers; integration tests can install responses to simulate status codes, headers (Retry-After, X-RateLimit-Reset, ETag, Link), and body JSON.

• For get_with_retries, write table-driven tests that feed 403/429 with:

  • Retry-After: 5
  • HTTP-date Retry-After
  • X-RateLimit-Remaining: 0 + X-RateLimit-Reset
  • 5xx with exponential backoff
    Assert: sleep/backoff behavior (mock time.sleep), early abort on budget, and caplog messages.

5. Probe/since/ETag logic

• probe_since_support and fetch_endpoint are the trickiest. Use responses to:

  • First page returns data earlier than since → detect “since ignored” path.
  • 304 with If-None-Match on first page → no new data.
  • Pagination via Link: <...page=2>; rel="next".
    Assert correct appends, sort order, and ETag file read/write.

6. Sorting & de-dup

• You dedupe by id then sort by updated_at_iso/created_at_iso. Create a small synthetic set to ensure stable ordering (oldest→newest) and overwrite by id works.

7. Logging and error coverage

• You already have rich logging. Use caplog to assert error and warning lines for: corrupt JSON renamed, file write failure, rate-limit abort, non-JSON page, server error, etc.

8. JSON/CSV schema checks

• Define a JSON Schema for a “comment record”. Validate the output list and a single record:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["id","owner","repo","source_endpoint","user_login","created_at_iso","raw"],
  "properties": {
    "id": {"type": ["integer","string"]},
    "owner": {"type":"string"},
    "repo": {"type":"string"},
    "source_endpoint": {"enum":["issues","reviews"]},
    "user_login": {"type":"string"},
    "created_at_iso": {"type":"string", "pattern":"^\\d{4}-\\d{2}-\\d{2}T.*Z$"},
    "updated_at_iso": {"type":["string","null"]},
    "reactions_total": {"type":"integer"},
    "reactions_breakdown": {"type":"object"},
    "raw": {"type":"object"}
  },
  "additionalProperties": true
}

• For CSV (if/when you emit), use frictionless (Table Schema) to assert columns and basic constraints.

9. Deterministic time

• Add freezegun (or time-machine) to pin datetime.now() in tests around meta writes and _parse_retry_after with HTTP-date.

10. Golden cassettes vs live

• Use vcrpy cassettes for “integration-like” tests of the GitHub endpoints (small pages). Keep them off by default in CI, enabled via an env flag, so forks don’t need secrets.

Minimal test layout (tailored to your files)

tests/
  unit/
    test_envless_import.py         # ensure importing module never exits
    test_utils_time.py             # _parse_iso/_iso/decrement_one_second/_parse_retry_after
    test_utils_usertrim.py         # trim_user/trim_comments
    test_records.py                # to_assessment_record/_from_review
    test_dedupe_sort.py            # last_seen_by_endpoint, sort key, id overwrite
    test_split_owner_repo.py
  http/
    test_get_with_retries.py       # 403/429/reset/date/5xx/abort paths (responses + caplog)
    test_probe_and_etag.py         # probe_since_support + 304/ETag
    test_fetch_endpoint_paging.py  # Link paging, users filter, write_etag on page1
  io/
    test_load_save_json.py         # corrupt file moved, atomic save
    test_etag_meta_paths.py        # read/write_etag, read_meta/write_meta
  schemas/
    data_record.schema.json
  integration/                     # optional; use vcrpy
    test_github_issues_vcr.py
    test_github_reviews_vcr.py

Example pytest snippets you’ll want

• Prevent import-time exit (until you refactor):

def test_module_import_without_token(monkeypatch):
    monkeypatch.setenv("GITHUB_TOKEN", "x")  # set before import
    import scripts.fetch_issue_comments as mod  # should not SystemExit

• Mock sleeps & logs in backoff tests:

def test_rate_limit_retry(monkeypatch, caplog, requests_mock):
    monkeypatch.setattr("time.sleep", lambda *_: None)
    # set up responses: 429 with Retry-After, then 200
    ...
    assert "Rate limited" in caplog.text

• JSON schema validation:

from jsonschema import validate
def test_record_schema(record, schema):
    validate(record, schema)

CI changes I’d make (in addition to your fetch workflow)

1. Add a dedicated Test workflow

• Run on PRs and pushes; matrix for 3.10–3.12.
• Install dev extras and run pytest with coverage gate; run ruff + mypy.
• Keep “live API” tests behind RUN_LIVE_API=true and separate job with repository secrets.

name: Tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix: { python-version: ['3.10','3.11','3.12'] }
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: ${{ matrix.python-version }} }
      - name: Install deps
        run: |
          python -m pip install -U pip
          pip install -e .[dev]
      - name: Lint & types
        run: |
          ruff check .
          mypy scripts
      - name: Run tests
        run: |
          pytest -q --cov=scripts --cov-report=xml --cov-fail-under=85

2. Pre-commit hook config

• Add ruff, black (if you want), and pytest --maxfail=1 -q as local hook if you like.

3. tox/nox (optional but nice)

• Mirror CI locally: envs for py310,py311,py312, a lint env, and a types env.

Package list (updated)

• Must: pytest, pytest-cov, pytest-mock, responses, vcrpy, hypothesis, jsonschema, frictionless, freezegun (or time-machine), ruff, mypy, pre-commit, tox (or nox).

Small refactors that pay off in tests

• Config loader: create load_config(os_environ) -> dataclass that resolves owner/repo/users/flags/output_path/token. Unit test it with monkeypatch.setenv.
• I/O boundary: keep ETag/meta IO behind tiny functions you already have; they’re easy to fake with tmp_path.
• HTTP boundary: all HTTP goes through get_with_retries → easier to test fetch behaviors by stubbing that function in higher-level tests.

Your existing YAML

• Keep your Fetch Issue Comments workflow as-is; add the Tests workflow above.
• Consider adding a scheduled run for fetch if you need periodic syncs; tests still run on PRs.

---

If you want, I can drop in a ready-to-use tests/ skeleton (a few real tests for _parse_retry_after, ETag 304 flow, and JSON schema validation) plus a pyproject.toml dev extras block and a tests.yml for CI.

[JasonUranta]

Instructions

1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
2. Blockers: "Difficulties or errors encountered."
3. Availability: "How much time will you have this week to work on this issue?"
4. ETA: "When do you expect this issue to be completed?"
5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."

1. Progress: Fed ChatGPT 5 the Python scripts, YAML files, and some of the JSON outputs of the ai-skills-accessor repo in order to get a full picture of what our testing suite should look like. Clarified some details regarding our ideal testing suite and had ChatGPT generate a full testing suite we could use as a model/example and potentially build off of.
2. Blockers: Give Chat a full picture of the repo required batching the data and providing background information
3. Availability: N/A
4. ETA: N/A
5. Pictures (if necessary): N/A
6. Link to ChatGPT Conversation: https://chatgpt.com/share/69095fea-7c68-800e-a826-f8f62943f475
7. Link to zipped testing Suite Folder: ai_skills_assessor_full_tests.zip

[chinaexpert1]

I'm leaving this open so Bonnie can see the status of the project.