fix: preserve styling and images in URL-ingested citation pages

[PadhiAjit-Microsoft]

Purpose

• This pull request enhances the ingestion pipeline to improve how HTML pages are stored and rendered for citation. The main change ensures that when an HTML page is ingested, it is modified to preserve its appearance and prevent scripts from running when viewed from a blob URL. Several robust tests have also been added to verify this behavior.

HTML snapshot preparation and security:

• Added the _prepare_html_snapshot function to inject a <base href> (for resolving relative URLs) and a strict Content Security Policy <meta> (to prevent script execution) into HTML pages before storage. This ensures that when the stored HTML is viewed, it renders with correct styling and images but cannot execute scripts.
• Modified ingest_url to apply _prepare_html_snapshot to HTML content only, leaving other file types (e.g., PDFs) unchanged. [1] [2]

Testing improvements:

• Added comprehensive tests for _prepare_html_snapshot, covering correct injection after <head> or <html>, proper escaping of the source URL, and ensuring non-HTML files are unmodified.
• Updated ingestion tests to check that HTML content is processed with _prepare_html_snapshot before upload, and non-HTML content remains untouched.

Imports and refactoring:

• Added necessary imports for html and ParserKey to support the new logic. [1] [2]

Does this introduce a breaking change?

• Yes
• No

How to Test

• Get the code

git clone [repo-address]
cd [repo-name]
git checkout [branch-name]
npm install

• Test the code

What to Check

Verify that the following are valid

• ...

Other Information