[BUG]: Web scraping fetch fallback ignores captureAs, so Raw HTML and CSS Query Selector blocks get Markdown

[marmar9615-cloud]

How are you running AnythingLLM?

All versions

What happened?

In an Agent Flow, a Web Scraping block set to "Raw HTML" returns Markdown whenever the collector cannot use Puppeteer and falls back to a plain fetch. A block set to "CSS Query Selector" fails on the same deployments with URL could not be scraped and no content was found. The selector runs against Markdown and matches nothing.

What I expected: the fallback returns the same format the Puppeteer path returns for the same captureAs value. "Raw HTML" gets the page's HTML. "CSS Query Selector" gets the selected element's HTML. "Text content only" keeps getting Markdown.

Code path, master at eb7df1e:

• server/utils/agentFlows/executors/web-scraping.js:23 maps querySelector to html, and lines 25-30 call CollectorApi.getLinkContent(url, captureMode) and then run the selector over the result.

• server/utils/collectorApi/index.js:294 posts { link, captureAs } to the collector's POST /util/get-link (collector/index.js:134-152). That route calls getLinkText (collector/processLink/index.js:33-41), which calls scrapeGenericUrl and then getPageContent in collector/processLink/convert/generic.js.

• The Puppeteer branch honors the flag (generic.js:176-177):

  if (captureAs === "html") return innerHTML;
  return htmlToMarkdown(innerHTML, link);

• The fetch fallback does not (generic.js:220-230). It always ends in return htmlToMarkdown(pageText, link);. The fallback has never checked captureAs. Before Turn HTML scraped sites to Markdown for better research #5742 it returned the fetched page for every mode. Since Turn HTML scraped sites to Markdown for better research #5742, first released in v1.14.0, it returns Markdown for every mode.

• With a selector, parseHTMLwithSelector finds zero elements in the Markdown and returns { success: false } (web-scraping.js:95-96), and the block throws at line 34.

Any throw from the Puppeteer loader takes this path, a launch failure or a navigation error included. #4816 was one report of it firing inside the Docker image on an ERR_REQUIRE_ESM crash. That crash was fixed in #4819 and the issue is closed, but the fallback it exposed is the code path above. The collector logs getPageContent failed to be fetched by puppeteer - falling back to fetch! right before it. So the same flow works on one deployment and fails on another, with no change to the flow itself.

Are there known steps to reproduce?

Through the product. To put any install on the fallback, point the collector at a Chromium that is not there and restart it. The Docker image sets PUPPETEER_EXECUTABLE_PATH=/app/chrome-linux/chrome (docker/Dockerfile:70), so running the container with -e PUPPETEER_EXECUTABLE_PATH=/nonexistent/chrome is enough. The collector then logs getPageContent failed to be fetched by puppeteer - falling back to fetch! for every URL.

1. Settings > Agent Skills > Agent Flows, create a flow.
2. Add a Web Scraping block. Set "URL to Scrape" to a page with a known element, for example https://example.com. Set "Capture Page Content As" to "Raw HTML". Run the flow.
3. The block's result is Markdown (# Example Domain ...), not the page's HTML.
4. Change "Capture Page Content As" to "CSS Query Selector" and set "Query Selector" to h1. Run again. The flow fails with "URL could not be scraped and no content was found."

Against the collector directly, without a browser. The script below serves a fixed page from 127.0.0.1, points PUPPETEER_EXECUTABLE_PATH at a missing binary so the loader throws, and calls the collector's real getLinkText(url, "html") and getLinkText(url, "text"). It then runs the same cheerio selector the CSS Query Selector mode runs. Save it as repro.js and run node repro.js from the repo root.

repro.js

// Save as repro.js and run from the repo root: node repro.js
// Points Puppeteer at a missing binary so getPageContent takes its fetch
// fallback, serves a fixed page from 127.0.0.1, and calls the collector's
// real getLinkText() once per capture mode. Then runs the same cheerio
// selector the Agent Flow "CSS Query Selector" mode runs on the html result.
const http = require("http");
const path = require("path");

const ROOT = process.cwd();
process.env.PUPPETEER_EXECUTABLE_PATH = "/nonexistent/chrome";
process.env.STORAGE_DIR ||= path.join(ROOT, "collector/storage");

const cheerio = require(path.join(ROOT, "server/node_modules/cheerio"));
const { getLinkText } = require(path.join(ROOT, "collector/processLink"));

const HTML = `<!doctype html><html><head><title>T</title></head><body>
<div id="target"><h1>Hello</h1><p>Some <a href="https://example.com">link</a></p></div>
</body></html>`;

const server = http.createServer((req, res) => {
  res.writeHead(200, { "Content-Type": "text/html" });
  if (req.method === "HEAD") return res.end();
  res.end(HTML);
});

server.listen(0, "127.0.0.1", async () => {
  const url = `http://127.0.0.1:${server.address().port}/page`;
  const asHtml = await getLinkText(url, "html");
  const asText = await getLinkText(url, "text");

  console.log("\n===== captureAs: 'html' =====");
  console.log(JSON.stringify(asHtml.content));
  console.log(
    "cheerio $('#target') count on the html-mode result:",
    cheerio.load(String(asHtml.content))("#target").length
  );

  console.log("\n===== captureAs: 'text' =====");
  console.log(JSON.stringify(asText.content));
  console.log(
    "text-mode result contains '<div id=\"target\">' ?",
    String(asText.content).includes('<div id="target">')
  );

  console.log(
    "\nhtml result identical to text result?",
    asHtml.content === asText.content
  );
  server.close();
});

On master (eb7df1e):

===== captureAs: 'html' =====
"T\n\n# Hello\n\nSome [link](https://example.com)"
cheerio $('#target') count on the html-mode result: 0

===== captureAs: 'text' =====
"T\n\n# Hello\n\nSome [link](https://example.com)"
text-mode result contains '<div id="target">' ? false

html result identical to text result? true

The served page was <div id="target"><h1>Hello</h1><p>Some <a href="https://example.com">link</a></p></div>.

The fix is one line in getPageContent's fallback: check captureAs the same way the Puppeteer branch does, and return the fetched page for html. Text mode keeps returning Markdown. With that change the html-mode result is the page's HTML, the selector count is 1, and the text-mode result is unchanged. PR to follow.

LLM Provider & Model (if applicable)

Not applicable. The block fails before any model call.

Embedder Provider & Model (if applicable)

Not applicable.