Skip to content

ch5 registry/CPU gauge, ch7 error.type, stable HTTP semconv opt-in, Linux scraping (ch3-11) - #71

Merged
soumendrak merged 24 commits into
mainfrom
fix/ch5-ch7-code
Nov 1, 2026
Merged

soumendrak merged 24 commits into
mainfrom
fix/ch5-ch7-code

Conversation

@soumendrak

@soumendrak soumendrak commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #70 (base fix/json-logging). Merge order: #67 → #68 → #69 → #70 → this.

What changed (audit items)

  • Item 8 (Ch5): removed REGISTRY.register(reader) from api_gateway.py and order_service.py. PrometheusMetricReader.__init__ already registers its own collector, so the extra call raised TypeError: 'NoneType' object is not iterable on every start, and the try/except hid it. setup_metrics() now just returns the MeterProvider.
  • Item 8 (Ch5): added the psutil gauge the article describes as the semconv system.cpu.utilization ObservableGauge. It uses unit 1 and reports psutil.cpu_percent() / 100, a 0–1 ratio, per the system metrics semconv. The article uses %, so the article PR has to match this. psutil was already a dependency.
  • Ch5 typing and tests: both services are fully annotated, the comments are gone, and mypy --strict passes. pytest checks 100% coverage of api_gateway.py and order_service.py. Dev deps: pytest, pytest-cov, mypy, types-psutil.
  • Item 5 (Ch7): error.type on failed spans, following Recording errors: "When the operation ends with an error, instrumentation … SHOULD set the error.type attribute".
    • In order_service.py, insert_order_record now sets it on DB and payment failures. call_external_api sets error.type where it used to set external.error_type.
    • The recovered cache-fallback path still has no error.type, as the spec requires for handled errors.
    • The ch7 README trace example is updated.
    • I left the gateway alone on purpose. Its errors sit on the auto-instrumented server span, and with stable semconv the FastAPI instrumentation sets error.type to the status code there ("500", "502", seen in Jaeger below). A manual value would just be overwritten.
  • Item 7 (Ch3–11): each Makefile now has export OTEL_SEMCONV_STABILITY_OPT_IN := http, so auto-instrumented spans use the stable HTTP names (HTTP semconv migration). Checked for impact first:
    • The ch8 production filter already matches both http.target and url.path.
    • Every other dashboard and alert query uses the app's own http_* metric labels, which this doesn't touch.
  • Found while verifying (Docker on Linux): Prometheus could not scrape the ch5–7 apps. Both targets were down, which breaks item 8's Prometheus checks.
    • host.docker.internal doesn't resolve on Linux without extra_hosts: host-gateway.
    • uvicorn binds 127.0.0.1 by default (settings).
    • Fix: added extra_hosts to the ch5–7 Prometheus service and --host 0.0.0.0 to run-order/run-gateway. Docker Desktop already resolves host.docker.internal, so nothing changes there.
  • The ch6 README table said exemplars come from context.get_current() passed to .record(). They actually come from prometheus_client's Histogram.observe(..., exemplar=...), and the README now says so (item 2).
  • CI: removed the unpinned-image check that CI, Python 3.13 per chapter, promtool test fix, docker compose v2 #67 added twice.

Skills applied

engineering-rules-for-soumen-s-repos, verify-changes-before-review, tdd (tests failed on the old code first), principle-prove-it-works, principle-fix-root-causes, principle-subtract-before-you-add (dropped the dead registration and the gateway error.type lines that the instrumentation overwrites), blast-radius (grep of every config, dashboard and alert before the semconv opt-in), no-comments, sourced-content-claims, pr-queue-hygiene.

Codex

Not used.

Evidence

New ch5 tests against the old services:

FAILED tests/test_api_gateway.py::test_setup_metrics_logs_no_registration_warning
FAILED tests/test_api_gateway.py::test_metrics_exposes_cpu_utilization_between_zero_and_one
2 failed, 3 passed, 1 warning

New ch7 span tests against the old order_service.py:

FAILED tests/test_error_spans.py::test_failed_db_insert_sets_error_type - Key...
FAILED tests/test_error_spans.py::test_failed_payment_sets_error_type_on_client_and_parent

After the change, with 10 repeated runs each:

ch5: Success: no issues found in 4 source files
     10/10 "5 passed"   api_gateway.py 54 0 100%, order_service.py 38 0 100%
ch7: Success: no issues found in 3 source files
     10/10 "8 passed"   logging_setup.py 100%; changed order_service.py lines 191, 255, 293 all covered

(Round 3: that Starlette httpx TestClient deprecation broke CI under pytest -W error once ch5 had tests; fixed in f3f7ad4, see below.)

ch3 spans in Jaeger, make run-zero with and without the opt-in (/api/v3/traces, HTTP keys only):

AFTER  ['http.request.method', 'http.response.status_code', 'http.route', 'network.peer.address', 'network.protocol.version', 'server.address', 'server.port', 'url.full', 'url.path', 'url.scheme']
BEFORE ['http.flavor', 'http.host', 'http.method', 'http.route', 'http.scheme', 'http.server_name', 'http.status_code', 'http.target', 'http.url', 'http.user_agent', 'net.host.port', 'net.peer.ip', 'net.peer.port']

ch5/6/7 stacks: docker compose up --wait, then make run-order run-gateway, then 20× /checkout. Before the host fix:

api-gateway down Get "http://host.docker.internal:8000/metrics": dial tcp 172.17.0.1:8000: connec

After:

===== ch5-metrics
api-gateway up / order-service up
system_cpu_utilization in Prometheus: api-gateway 0.008
jaeger services {"services":["order-service","api-gateway"]}
app logs: 0 tracebacks, 0 "failed to register"
===== ch6-three-pillars
api-gateway up / order-service up
8 exemplars; first labels: {'span_id': '3179bb01f95827c3', 'trace_id': 'adaa3b8ab8e3212bfd3e9182ad41e41a'}
jaeger services {"services":["api-gateway","order-service"]}
===== ch7-error-handling
api-gateway up / order-service up
9 exemplars; first labels: {'span_id': 'c23addc3c3590ce5', 'trace_id': 'e66f0ae08e7794276d1a8c481e0d0713'}
jaeger services {"services":["order-service","api-gateway"]}

ch7 error spans in Jaeger after 20 checkouts, as (span, status, error.type, has external.error_type):

9 ('insert_order_record', ERROR, 'DatabaseError', False)
1 ('external.stripe.charge', ERROR, 'ConnectionError', False)
1 ('insert_order_record', ERROR, 'ConnectionError', False)
9 ('GET /checkout', ERROR, '500', False)   <- set by the FastAPI instrumentation

302 changed lines (+177/-125) against fix/json-logging, excluding tests and uv.lock (round 4 head cfafaba; git diff --numstat origin/fix/json-logging...HEAD). The same count against main + #70 merged: +177/-125.

Review round 3 (heads 85adc47, f3f7ad4)

Fixed with normal commits on top of a normal merge of fix/json-logging (c952641). Nothing was force-pushed or rebased.

Reviewer item Change Commit
Negative test that the cache-fallback span is not an error, plus a mutant test_cache_fallback_span_is_not_an_error: source cache, status UNSET, no error.type, product.source=cache, events [exception, cache_fallback] 85adc47
Drop the ch7 mypy ignore_errors override and fix the types Override removed from pyproject.toml; order_service.py fully typed (response_model=None on the union-returning /orders, a typed render_openmetrics boundary for the untyped prometheus_client.openmetrics generate_latest). No ignore_errors, type: ignore or noqa left in any chapter dfc0cc7
/metrics coverage after the retype test_metrics_endpoint_renders_openmetrics 85adc47
Codex-bot P2: TestClient deprecation fails ch5 under -W error Added httpx2>=2.16.0 to ch5 dev deps, same as ch4 in #68. CI on 85adc47 failed exactly this way (ModuleNotFoundError: No module named 'httpx2' → StarletteDeprecationWarning) f3f7ad4

Mutation check (ch7 @ f3f7ad4, uv run pytest -W error)

unmutated: 13 passed
mutant: fallback adds span.set_attribute("error.type", type(e).__name__) -> FAILED test_cache_fallback_span_is_not_an_error (1 failed, 12 passed)
mutant: fallback adds span.set_status(StatusCode.ERROR, str(e))          -> FAILED test_cache_fallback_span_is_not_an_error (1 failed, 12 passed)

ch7 mypy --strict: Success: no issues found in 3 source files with the override gone. ch5 @ f3f7ad4: 5 passed, coverage 100%, mypy strict clean, ruff check clean, under pytest -W error.

Live ch7 error.type (stack @ 85adc47, Jaeger /api/v3/traces, order-service stopped for the last requests)

/metrics content-type: application/openmetrics-text; version=1.0.0; charset=utf-8 200
with order-service down: /products/widget -> 503, /checkout -> 503
api-gateway  GET /checkout              SERVER ERROR error.type 500 (15) / 502 (8) / 503 (1), each with an exception event
api-gateway  GET /products/{product_id} SERVER ERROR error.type 503 (1)  <- JSONResponse(503), no raise
api-gateway  POST / GET                 CLIENT ERROR error.type 500, 502, ConnectError
order-service POST /orders              SERVER ERROR error.type 500 (15) / 502 (8)
order-service insert_order_record       ERROR error.type DatabaseError (15) / ConnectionError (8)
order-service fetch_product with cache_fallback,exception events (14): status UNSET, error.type None
  • Disagreement: there is no "/products 502". In the code, the 502 is the /checkout → /orders payment failure. /products/{id} only returns 503 (order-service unreachable), and its server span carries error.type="503".
  • Finding, not fixed here: stripe double exception event Fixed in round 4 (cfafaba), see below.
  • With OTEL_SEMCONV_STABILITY_OPT_IN=http, auto-instrumentation already sets error.type to the status code on every 5xx server span. A FastAPI TestClient check confirms it overwrites a manually set value (Custom → "500"), and marks the route span ERROR whether the handler raises or returns JSONResponse(500). Only raise adds an exception event to the route span.

Boots @ 85adc47 (docker compose up, make run-order run-gateway, traffic)

ch9:  jaeger 2.22.0 healthy, collector 0.164.0 healthy, targets otel-collector:8889 up / :8888 up,
      otelcol_exporter_sent_spans otlp_grpc/jaeger 104, Jaeger services [api-gateway, order-service],
      JSON log lines order 110 ok / 0 invalid, gateway 80 ok / 0 invalid
      (0 otel_gateway_requests_total series: the run ended before the SDK's first 60 s metric export; ch9 has no Grafana)
ch10: grafana 13.3.0 db ok, jaeger 2.20.0 healthy, collector 0.164.0 healthy, both targets up,
      otel_gateway_requests_total 15 series (exported_job api-gateway), 18 exemplars with trace_id/span_id,
      datasources Prometheus PBFA97CFB590B2093 -> exemplar link PC9A941E8F2E49454 (Jaeger) both OK,
      dashboard red-dashboard, JSON log lines 150/0 and 100/0

Still open (not in this PR)

  • ch8–11 error.type follow-up: the same insert_order_record / call_external_api error.type changes for ch8–11. (The stripe double-exception fix for ch8–11 is now in this PR.)
  • Pre-existing, untouched: ruff format --check wants to wrap one unchanged line in ch5 api_gateway.py (request_counter.add(0, ...)). CI does not run ruff format.

Skills applied (round 3)

learn-from-review-rejections, engineering-rules-for-soumen-s-repos, verify-changes-before-review, principle-prove-it-works, principle-fix-root-causes, principle-test-behavior-not-implementation, principle-type-system-discipline, no-comments, pr-queue-hygiene, pre-delivery-checks.

Codex

Not used. The chatgpt-codex-connector review bot comments on this repo by itself; I fixed its finding and did not use Codex to write anything.

Review round 4 (rejected at f3f7ad4: stripe double exception event) → head cfafaba

New commits only. Nothing was force-pushed or rebased. Commits:

Reviewer item Change Commit
Tests first: stripe span event names == ['exception'] on a ConnectionError, failing at f3f7ad4 ch7 test_failed_payment_records_one_exception_event (event names == ["exception"], exception.type == "ConnectionError"). ch8–11: new tests/test_external_call.py (byte-identical in all four), with test_failed_external_call_records_one_exception_event (events == ["exception"], exception.type, status ERROR) and test_successful_external_call_records_no_events 19a8918
Fix with record_exception=False or drop the manual call, whichever matches the README record_exception=False on the external.<service>.<op> CLIENT span in call_external_api. The README teaches the manual span.record_exception(e) and "an exception Event", so the manual call stays and the SDK's automatic second recording goes cfafaba
Grep ch7 and every other chapter, fix each in the owning PR AST scan (start_as_current_span without record_exception=False whose body calls span.record_exception and re-raises): found only call_external_api in ch7, ch8, ch9, ch10 and ch11. No open PR other than this one touches those order_service.py files (#69 is merged, #70 is logging-only), so all five are fixed here. After the fix the scan finds nothing cfafaba
README matches ch7 README stripe trace example now shows events: exception (exactly one), plus a Key Concepts row "Record once, then re-raise" that explains why record_exception=False is there cfafaba
Merge #70 into this branch, keep CI green 1f18f2d (merge commit). CI on cfafaba: see Checked line 1f18f2d

ch8–11 order_service.py were untyped. Importing them from the new tests pulled them into CI's mypy --strict, so they are now fully annotated:

  • dict[str, Any] parameters and Callable[..., Awaitable[Any]];
  • return types on every handler;
  • response_model=None on the union-returning /orders and /ask, the same as ch7.

Tests fail first

At 19a8918 (tests only, the f3f7ad4 code):

ch7  AssertionError: assert ['exception', 'exception'] == ['exception']   1 failed, 14 passed
ch8  AssertionError: assert ['exception', 'exception'] == ['exception']   1 failed, 11 passed
ch9  AssertionError: assert ['exception', 'exception'] == ['exception']   1 failed, 11 passed
ch10 AssertionError: assert ['exception', 'exception'] == ['exception']   1 failed, 11 passed
ch11 AssertionError: assert ['exception', 'exception'] == ['exception']   1 failed, 11 passed

At cfafaba: ch7 15 passed, ch8–11 12 passed each. Coverage is 100% and mypy --strict is clean in all five. ruff check reports nothing new compared with f3f7ad4.

Mutation check (cfafaba, PYTHONDONTWRITEBYTECODE=1, __pycache__ cleared before each run)

Mutant ch7 ch8 ch9 ch10 ch11
record_exception=False removed (double event) killed killed killed killed killed
record_exception=True (double event) killed killed killed killed killed
manual span.record_exception dropped as well (zero events) killed killed killed killed killed
success path never sets status survives killed killed killed killed
span.set_status(ERROR) dropped survives survives survives survives survives
  • set_status dropped is an equivalent mutant. set_status_on_exception=True (the SDK default) sets ERROR anyway when the re-raised error leaves the with block, so no test can tell them apart.
  • The ch7 success-path survivor is pre-existing. That line is unchanged in this PR and ch7 has no success-status test. ch8–11 have one, via test_successful_external_call_records_no_events.

Live ch7 stack (Jaeger /api/v3/traces, events per span)

f3f7ad4-equivalent code (85adc47):

external.stripe.charge | CLIENT | ERROR | None | ConnectionError | exception,exception : 8

cfafaba:

external.stripe.charge | CLIENT | ERROR | None | ConnectionError | exception : 4
external.stripe.charge | CLIENT | UNSET | None | None |  : 23
insert_order_record | INTERNAL | ERROR | None | ConnectionError | exception : 4
insert_order_record | INTERNAL | ERROR | None | DatabaseError | exception : 10

Follow-up, not fixed here

  • With set_status_on_exception=True, the SDK replaces the manual status description (stripe API timeout) with ConnectionError: stripe API timeout when the error leaves the span. That is harmless, so it is left as is. It could be changed with set_status_on_exception=False if the manual description matters.

Skills applied (round 4)

learn-from-review-rejections, tdd (tests committed and shown failing before the fix), engineering-rules-for-soumen-s-repos, verify-changes-before-review, principle-prove-it-works, principle-fix-root-causes, principle-test-behavior-not-implementation, principle-type-system-discipline, no-comments, pr-queue-hygiene, pre-delivery-checks.

Codex NOT used.

Checked: at cfafaba, the new tests fail at 19a8918 and pass at cfafaba in ch7–11; the AST scan finds no double-record left; the event-count mutants are killed in all five chapters; the live ch7 stripe ERROR spans carry one exception event; git grep finds no noqa, type: ignore, pragma: no cover or ignore_errors; CI on cfafaba is green (16/16 checks pass).

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-01T19:24:11.106584Z cfafaba New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 85adc47a8b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


import httpx
import pytest
from fastapi.testclient import TestClient

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Handle the TestClient deprecation under CI's warning policy

With the locked Starlette version, these new TestClient-based tests emit a deprecation warning, as also shown in the commit's test evidence. However, the Python job in .github/workflows/ci.yml runs uv run pytest -W error, so that warning becomes an error and the newly enabled chapter 5 test suite fails CI even though ordinary pytest passes. Use the supported client API or narrowly filter this dependency warning while retaining the existing warning policy.

Useful? React with 👍 / 👎.

@soumendrak

Copy link
Copy Markdown
Owner Author

@soumendrak-reviewer-bot review

@soumendrak-reviewer-bot soumendrak-reviewer-bot left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REJECT at f3f7ad41099d295b93b6310a2b1b4863b9a5b05c. CRITICAL (edits .github/workflows/ci.yml). Round 3. Scope: the delta 1a67a0f..f3f7ad4 (merge c952641, dfc0cc7, 85adc47, f3f7ad4) plus regressions, and one finding I missed earlier, labelled as such.

CURRENT STATE. Everything from the earlier rounds is fixed, and I re-ran it at this head:

  • Merge c952641 has exactly the tree of a fresh merge of 1a67a0f and bc3d49b (957ab0f). Nothing extra came in with it.
  • The ch5 regression from 85adc47 is gone. The CI python job body, run locally for all 11 chapters, exits 0 on every one. ch5 under pytest -W error gave 5 passed 10 times out of 10, with api_gateway.py and order_service.py at 100%. mypy --strict is clean and uv lock --check passes.
  • ch7: the ignore_errors override is gone, mypy --strict reports Success: no issues found in 3 source files, and the run gives 13 passed. The 5 mutants (error.type on the fallback, ERROR status on the fallback, dropping record_exception, renaming the cache_fallback event, and plain-text /metrics) are all killed. The CONTROL passes.
  • CI has 16 of 16 check runs green. The Codex P2 is fixed.

One defect is still there, in the except block this PR edits: every failed Stripe call records its exception twice.

EXPECTED STATE. Each failed external.stripe.charge span carries exactly one exception event, a test fails if a second one comes back, and CI is green at the new head.

Must-fix

  1. MISSED EARLIER FINDING (it was in the round-1 diff): call_external_api records the exception twice. Location: chapters/ch7-error-handling/order_service.py:177-198. The manual span.record_exception(e) is at :191, followed by a re-raise at :195. start_as_current_span keeps its default record_exception=True.
    • Proof: drive create_order through an in-memory exporter, with the DB stubbed OK and the Stripe charge raising ConnectionError. The result is external.stripe.charge StatusCode.ERROR ['exception', 'exception']. insert_order_record has ['exception']. The current 13 tests still pass, so nothing guards this. The PR body reports the same thing from Jaeger: 8 of 8 failures had two events.
    • WHY: the SDK's context manager records any exception that leaves the with block. Recording it by hand and then re-raising stores it a second time. A chapter whose README says record_exception "adds an exception Event" then shows learners two identical stack traces on every failed call, and doubles the event volume. That teaches the wrong lesson in the one chapter about error recording. This PR is the one editing that block for error.type, so deferring it leaves a known defect in code it changes.
    • CHECK: add a test asserting [e.name for e in span_named("external.stripe.charge").events] == ["exception"]. It must fail at f3f7ad4. Then fix the double recording: either pass record_exception=False to that start_as_current_span, or drop the manual call. I checked the first option: the probe prints ['exception'] and all 13 tests still pass. The test must pass after the fix.

Merge note: squash this PR, because 8fe987d (six .coverage files) is in its history. #70 is still REJECT at bc3d49b, so after #70 changes, this branch needs it merged in again and the verdict re-checked.

Any new commit voids this verdict.

@soumendrak-reviewer-bot soumendrak-reviewer-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE @ cfafaba (round 4, CRITICAL: ci.yml)

Prior must-fix FIXED:

  • Commit 19a8918 fails in ch7-ch11 with ['exception','exception'] == ['exception'], and cfafaba passes (ch7 15/15, ch8-ch11 12/12 each).
  • An in-memory exporter, run 3 times per chapter, shows exactly one exception event per failure, with status ERROR and description "ConnectionError: stripe API timeout".
  • An AST scan finds no other record-and-reraise inside a span in ch7-ch11.

The claim that the "set_status dropped" mutant is equivalent is confirmed in two ways:

The "success path never sets status" line (order_service.py:186-189) dates from 490846c and was not touched by this PR, so it is out of round scope.

mypy --locked passes for ch5-ch11, the README matches the behaviour, the size reproduces at +177/-125, and CI is 16/16. Squash-merge, after #70 is merged.

@soumendrak
soumendrak changed the base branch from fix/json-logging to main November 1, 2026 20:13
@soumendrak
soumendrak merged commit 15beeed into main Nov 1, 2026
16 checks passed
@soumendrak
soumendrak deleted the fix/ch5-ch7-code branch November 1, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants