refactor(transformers): drop observability diagnostics and disambiguate readability decline

Content transformers no longer emit observability diagnostics: the readability and mdream char-count summaries and the empty_output flag are removed. mdream returns empty markdown as a transformed empty body, so a content-free page fails the run via the empty-body rollup. Readability's empty-article decline is renamed parse_empty -> output_empty, distinct from the no-document parse_empty case. Remove the now-inert emitDiagnostics flags from the shipped pipelines and sync the docs.

Author: sousekd

config/llm-context-loader.yaml

@@ -125,14 +125,12 @@ pipelines:
         config:
           transformer: readability-default
           target: text/html
-          emitDiagnostics: true
       - type: transform
         name: convert
         concurrencyGroup: process
         config:
           transformer: mdream-convert
           target: text/markdown
-          emitDiagnostics: true
       - type: truncate
         name: truncate
         config:
@@ -217,14 +215,12 @@ pipelines:
         config:
           transformer: readability-default
           target: text/html
-          emitDiagnostics: true
       - type: transform
         name: convert
         concurrencyGroup: process
         config:
           transformer: mdream-convert
           target: text/markdown
-          emitDiagnostics: true
       - type: capture-urls
         name: capture_source_urls
         concurrencyGroup: process

docs/CUSTOMIZATION.md

@@ -203,7 +203,7 @@ config:
 
 Supports text `text/html` or `application/xhtml+xml` bodies targeting `text/html`. The incoming title is preserved; when absent, the extracted article title is used instead.
 
-Decline reasons are `not_readerable` and `parse_empty`. Runtime errors propagate to the orchestrator; the failed step has no effect, so downstream steps continue from the prior body and the run rolls up as `degraded`.
+Decline reasons are `not_readerable`, `parse_empty`, and `output_empty`. Runtime errors propagate to the orchestrator; the failed step has no effect, so downstream steps continue from the prior body and the run rolls up as `degraded`.
 
 ### `mdream`
 
@@ -349,7 +349,7 @@ Applies a named content transformer to the current body. The step resolves `tran
 | `transformer`     | string           | —       | Name of a configured `contentTransformers` instance. Required.                                                 |
 | `target`          | string           | —       | Media type the transformer must produce, for example `text/markdown`. Required.                                |
 | `onUnsupported`   | `skip` \| `fail` | `skip`  | What to do when the transformer does not support the current body (wrong source media type or representation). |
-| `onDeclined`      | `skip` \| `fail` | `skip`  | What to do when the transformer returns `declined` (e.g. `not_readerable`, `parse_empty`).                     |
+| `onDeclined`      | `skip` \| `fail` | `skip`  | What to do when the transformer returns `declined`.                                                            |
 | `emitDiagnostics` | bool             | `false` | When enabled, transformer-reported diagnostics are surfaced as child nodes in the step report.                 |
 
 The step skips with `no_body` when there is no body. When the transformer does not support the current body it skips with `unsupported` (or fails with `onUnsupported: fail`). When the transformer returns `declined`, the step skips with that reason (or fails with `onDeclined: fail`). A transform aborted by the step timeout fails with `timeout`. If the transformer returns a body that does not match `target`, the step fails with `wrong_output_type`.

docs/agents/extension-authoring.md

@@ -74,7 +74,7 @@ The result type is a discriminated union:
 
 `supports` gates `transform`: the step only invokes a matching transformer, so `transform` may throw `InternalError` for inputs that bypass the gate. Use `declined` for deliberate "not suitable" decisions (e.g. `isProbablyReaderable` returned false).
 
-Built-in examples: `src/builtins/content-transformers/readability/` (article HTML extraction, outputs `declined`), `src/builtins/content-transformers/mdream/` (HTML to markdown, always `transformed`).
+Built-in examples: `src/builtins/content-transformers/readability/` (article HTML extraction, can return `declined`), `src/builtins/content-transformers/mdream/` (HTML to markdown, always `transformed`).
 
 ### LLM Providers
 

docs/agents/smoke-testing.md

@@ -104,7 +104,7 @@ For the OWUI surface the per-URL time column is the **batch** time shared by eve
 
 When the script output is not enough, read the server logs: pipeline start/finish lines carry the URL, outcome, and duration, and step warnings carry a `reason`. To dig into a single URL, use `inspect-suspects.ps1`.
 
-A `result="failed"` with `final_length=0` does not imply a pipeline bug by itself. Check the step entries in the footer: was the fetch step `ok`? Did all steps that should run actually run? A footer step showing `empty_output` is normal for 404 pages, paywalled content, or JavaScript-rendered SPAs — the pipeline cannot salvage those. Look at each step's `status` and `reason` before deciding something is wrong.
+A `result="failed"` with `final_length=0` does not imply a pipeline bug by itself. Check the step entries in the footer: was the fetch step `ok`? Did all steps that should run actually run? An empty final body is normal for 404 pages, paywalled content, or JavaScript-rendered SPAs — the deterministic transformers convert them to empty markdown and the pipeline cannot salvage those. Look at each step's `status` and `reason` before deciding something is wrong.
 
 ## Common Traps
 

src/builtins/content-transformers/mdream/mdream-transformer.ts

@@ -5,6 +5,8 @@
  * passes the source URL as the mdream `origin` so relative links and images
  * resolve to absolute URLs. The `minimal` preset additionally isolates main
  * content and filters boilerplate; `clean` applies link and whitespace cleanup.
+ * Empty conversion output is returned as a valid empty body, not a decline, so
+ * a content-free page (e.g. a script-only SPA shell) fails the run honestly.
  */
 
 import { htmlToMarkdown, withMinimalPreset, type MdreamOptions } from "@mdream/js";
@@ -13,7 +15,6 @@ import { InternalError } from "../../../shared/errors.js";
 import { isHtmlMediaType, mediaTypes } from "../../../shared/media-types.js";
 
 import type {
-  ContentTransformDiagnostic,
   ContentTransformRequest,
   ContentTransformResult,
   ContentTransformer
@@ -54,15 +55,10 @@ export class MdreamTransformer implements ContentTransformer {
       throw new InternalError("mdream transformer requires a text body", "mdream_non_text_body");
 
     const markdown = htmlToMarkdown(body.content, this.buildOptions(input.url)).trim();
-    const diagnostics: ContentTransformDiagnostic[] = [
-      { code: "mdream", message: `${body.content.length} html chars -> ${markdown.length} markdown chars` }
-    ];
-    if (!markdown) diagnostics.push({ code: "empty_output" });
 
     return {
       outcome: "transformed",
-      body: { kind: "text", mediaType: mediaTypes.markdown, content: markdown, title: body.title },
-      diagnostics
+      body: { kind: "text", mediaType: mediaTypes.markdown, content: markdown, title: body.title }
     };
   }
 

src/builtins/content-transformers/readability/readability-transformer.ts

@@ -15,11 +15,7 @@ import { parseHTML } from "linkedom";
 import { InternalError } from "../../../shared/errors.js";
 import { isHtmlMediaType, mediaTypes } from "../../../shared/media-types.js";
 
-import type {
-  ContentTransformDiagnostic,
-  ContentTransformResult,
-  ContentTransformer
-} from "../../../contracts/extensions/content-transformer.js";
+import type { ContentTransformResult, ContentTransformer } from "../../../contracts/extensions/content-transformer.js";
 import type { BodyContent } from "../../../contracts/pipeline/context.js";
 import type { Logger } from "../../../shared/logger.js";
 import type { ReadabilityTransformerConfig } from "./readability-transformer-config.js";
@@ -70,16 +66,7 @@ export class ReadabilityTransformer implements ContentTransformer {
       return { outcome: "declined", reason: "not_readerable" };
 
     const article = new Readability(doc, { maxElemsToParse: this.config.maxElements || undefined }).parse();
-    if (!article || !article.content) return { outcome: "declined", reason: "parse_empty" };
-
-    const originalLength = body.content.length;
-    const articleLength = article.content.length;
-    const diagnostics: ContentTransformDiagnostic[] = [
-      {
-        code: "readability",
-        message: `${originalLength} html chars -> ${articleLength} article chars`
-      }
-    ];
+    if (!article || !article.content) return { outcome: "declined", reason: "output_empty" };
 
     return {
       outcome: "transformed",
@@ -88,8 +75,7 @@ export class ReadabilityTransformer implements ContentTransformer {
         mediaType: mediaTypes.html,
         content: article.content,
         title: body.title || (article.title ?? undefined)
-      },
-      diagnostics
+      }
     };
   }
 }

tests/builtins/content-transformers/mdream/mdream-transformer.test.ts

@@ -82,7 +82,7 @@ describe("MdreamTransformer.transform", () => {
     expect(result.body.content).toContain("](https://example.com");
   });
 
-  it("emits a summary diagnostic and flags empty output", async () => {
+  it("returns a transformed empty body when conversion yields no markdown", async () => {
     const transformer = makeTransformer();
     const empty = await transformer.transform(
       {
@@ -94,8 +94,9 @@ describe("MdreamTransformer.transform", () => {
     );
     expect(empty.outcome).toBe("transformed");
     if (empty.outcome !== "transformed") throw new Error("Expected transformed outcome");
-    expect(empty.diagnostics?.some(diagnostic => diagnostic.code === "empty_output")).toBe(true);
-    expect(empty.diagnostics?.some(diagnostic => diagnostic.code === "mdream")).toBe(true);
+    expect(empty.body.kind).toBe("text");
+    if (empty.body.kind !== "text") throw new Error("Expected text body");
+    expect(empty.body.content).toBe("");
   });
 
   it("throws when given a non-text body", async () => {

tests/builtins/content-transformers/readability/readability-transformer.test.ts

@@ -95,7 +95,7 @@ describe("ReadabilityTransformer.transform", () => {
     expect(result.body.title).toBe("Page Title");
   });
 
-  it("adds a readability diagnostic with char counts", async () => {
+  it("transforms article HTML without diagnostics", async () => {
     const transformer = makeTransformer();
     const body: BodyContent = {
       kind: "text",
@@ -107,7 +107,7 @@ describe("ReadabilityTransformer.transform", () => {
 
     expect(result.outcome).toBe("transformed");
     if (result.outcome !== "transformed") return;
-    expect(result.diagnostics?.some(d => d.code === "readability")).toBe(true);
+    expect(result.diagnostics).toBeUndefined();
   });
 
   it("declines not_readerable for short boilerplate HTML", async () => {