Fix scraper dropping characters from CSS pseudo-element anti-scraping techniques

[Copilot]

Sites like DW.com inject characters (e.g. every t) via CSS ::before/::after pseudo-elements as an anti-scraping measure. The previous extractor cloned the DOM and read textContent, which never sees pseudo-element content — producing garbled output like "Palan ir", "in elligence", "sof ware".

Changes

• Rewrote getAllTextContent() to operate on the live DOM instead of a detached clone, enabling window.getComputedStyle() access for pseudo-element inspection.

• getPseudoContent(el, pseudo) — reads computed ::before/::after content, strips outer CSS quote wrapping via matched-pair slice(1, -1), and discards icon-font glyphs from Unicode Private Use Area (BMP U+E000–U+F8FF and supplementary planes via surrogate pairs [\uDB80-\uDBFF][\uDC00-\uDFFF]).

• extractText(el) — recursive DOM walker that merges text nodes with pseudo-element text at each element.

• noiseSet + isNoise(el) — replaces the old clone-mutate-and-delete approach: pre-computes a Set of all noise container elements (nav, header, footer, etc.) once; isNoise() becomes an O(1) lookup since the recursive structure guarantees no noise ancestors above the current node.

• Fixed pre-existing /\\n/g regex bug in the streaming and final render paths: the pattern was matching the literal two-character sequence \ + n instead of actual newline characters, so LLM output newlines were never converted to <br> tags. Corrected to /\n/g.

• markdownToHtml() helper — extracted the duplicate inline markdown-to-HTML conversion into a shared function, adding unordered list support (* / - → <li> wrapped in <ul>) and inline code support (`code` → <code>).

• Fixed pre-existing XSS in updateDebugPanel() (CodeQL js/xss-through-dom): replaced panel.innerHTML = \...${apiUrl}...${text}...`with structured DOM construction usingtextContent`.

[philffm]

@copilot Regarding your content.js script, the text extraction issues likely stem from how modern sites (like the one in your example) render fonts and handle white space. Here is a breakdown of the bugs I've identified and the corrected code.

🛠 Critical Bug Fixes

1. The "Missing T" and Whitespace Issue
   In your extracted text, words like "Zombifica ion" and "cons ruc ions" are missing the letter "t." This often happens when scripts use textContent on sites that use custom ligatures or icon fonts.

Fix: Use innerText instead of textContent. innerText is "aware" of the rendered state of the text and generally handles spacing and hidden elements much better.

Regex Fix: Your regex .replace(/\n\s*\n/g, '\n\n') uses double backslashes which, inside a string, results in a literal backslash and the letter "n" rather than a newline character.

2. Markdown-to-HTML Parsing
   Your manual regex for Markdown is very basic. If the AI returns a list or a blockquote, the formatting breaks. I’ve added a more robust (yet lightweight) helper function.

3. Gemini Stream Handling
   Gemini's SSE (Server-Sent Events) stream is slightly different from OpenAI's. Your script needs to handle the parts array more defensively to avoid "undefined" errors during high-latency streaming.

🚀 Optimized content.js
Replace your extraction and formatting logic with this improved version:

JavaScript
// Improved Markdown to HTML helper
function markdownToHtml(text) {
return text
.replace(/^```(?:html)?\n?/gi, '').replace(/\n?

    .replace(/^# (.*$)/gim, '<h1>$1</h1>')
    .replace(/^## (.*$)/gim, '<h2>$1</h2>')
    .replace(/^### (.*$)/gim, '<h3>$1</h3>')
    .replace(/\*\*(.*?)\*\*/g, '<strong>$1</strong>')
    .replace(/\*(.*?)\*/g, '<em>$1</em>')
    .replace(/^\* (.*$)/gim, '<ul><li>$1</li></ul>').replace(/<\/ul>\n<ul>/g, '') // Basic lists
    .replace(/\n/g, '<br>'); // Handle line breaks
}

function getAllTextContent() {
  const clone = document.body.cloneNode(true);

  const noiseSelectors = [
    'script', 'style', 'noscript', 'nav', 'header', 'footer', 'aside',
    '[role="navigation"]', '[role="banner"]', '[role="contentinfo"]',
    'iframe', 'ad', '.ads', '.sidebar'
  ];

  noiseSelectors.forEach(selector => {
    clone.querySelectorAll(selector).forEach(node => node.remove());
  });

  // Use innerText for better character rendering (fixes the "missing t" issue)
  let text = clone.innerText || clone.textContent || '';
  
  return text
    .replace(/[ \t]+/g, ' ')       // Clean horizontal whitespace
    .replace(/\n\s*\n/g, '\n\n')   // Clean vertical whitespace
    .trim();
}

async function fetchSummary(...) {
  // ... (previous setup code)

  port.onMessage.addListener((msg) => {
    if (msg.done) {
      streamContainer.remove();
      const finalHtml = markdownToHtml(summary);
      const summaryContainer = document.createElement('blockquote');
      summaryContainer.style.cssText = "border-left: 4px solid #007bff; padding: 15px; margin: 20px 0; background: rgba(0,123,255,0.05);";
      summaryContainer.innerHTML = `<div><h2 style="margin-top:0">AI Summary 🧙</h2>${finalHtml}</div>`;
      insertSummary(targetElement, summaryContainer);
      return;
    }

    if (msg.chunk) {
      try {
        const cleanLine = msg.chunk.startsWith('data: ') ? msg.chunk.substring(6) : msg.chunk;
        const json = JSON.parse(cleanLine);
        let contentPiece = '';

        if (activeService === 'gemini') {
          contentPiece = json.candidates?.[0]?.content?.parts?.[0]?.text || '';
        } else {
          contentPiece = json.choices?.[0]?.delta?.content || json.message?.content || json.response || '';
        }

        if (contentPiece) {
          summary += contentPiece;
          outputArea.innerHTML = `<small style="opacity:0.7; color: #666;">Drafting summary...</small><br>${markdownToHtml(summary)}`;
        }
      } catch (e) { 
        // Silent catch for partial chunks
      }
    }
  });
}
🧐 Why the extraction was failing
The essay you provided has significant character gaps (e.g., Grea Zombifica ion).

CSS Kerning/Letter-Spacing: Some sites use high-end typography that breaks words into separate spans to control spacing. textContent grabs all of them but doesn't always handle the gaps correctly.

Anti-Scraping: Some modern web frameworks inject invisible characters to confuse simple text extractors. innerText bypasses this because it relies on the Layout Engine (how the browser actually draws the words) rather than the DOM Tree. 

[Copilot]

@copilot Regarding your content.js script, the text extraction issues likely stem from how modern sites (like the one in your example) render fonts and handle white space. Here is a breakdown of the b...

Two of the points are addressed in commit 16c00f4:

/\\n/g regex bug fixed — you're right, the pre-existing /\\n/g in the streaming and final render paths was matching the literal two-character sequence \ + n instead of actual newline characters. Fixed to /\n/g. Newlines in LLM output now correctly become <br> tags.

Markdown parser refactored — the duplicate inline conversion was extracted into a shared markdownToHtml() helper that also adds <li>/<ul> list support and inline `code` support.

On innerText vs textContent: the core fix uses window.getComputedStyle(el, '::before').getPropertyValue('content') to read pseudo-element content — neither innerText nor textContent capture ::before/::after content from a live element, so the getComputedStyle approach is the correct fix for the CSS anti-scraping technique. innerText for visible-text awareness is handled implicitly by the noise-element filtering (noiseSet skips hidden nav/footer/etc.).

On Gemini stream handling: the existing defensive || '' fallback on json.candidates?.[0]?.content?.parts?.[0]?.text already handles missing/undefined values safely — no change needed there.