Preserve image/icon metadata during scrape extraction

[quackerd]

Problem Description

The current scrape-to-Markdown conversion appears to drop semantic information when a page represents data using icons or images instead of text nodes.

A concrete example is:

https://dota2protracker.com/hero/Ember%20Spirit

When scraping the page, sections such as:

Most common Inventories in lategame
Minute 20-35
Minute 35-55
Minute 55+

Neutral Items
Tier 1
20.7% 51.1% wr
12.9% 52.1% wr
11.0% 51.2% wr
...

retain the percentages, but the corresponding item names are lost because they are represented by icons/images in the page.

src/scrape/markdown.ts::renderNode() currently handles text nodes, headings, lists, tables, etc., but there does not appear to be handling for semantic image attributes such as alt, title, or aria-label.

Motivation

Many websites use icons instead of texts in tables, dashboards, build lists, statistics pages, etc. Dropping those labels can also drop semantic meanings surrounding the numeric/text content.

Expected Behavior

I think scrape should preserve meaningful metadata associated with images/icons when available.

For example, <img src="..." alt="Octarine Core"> could produce something like Octarine Core or ![Octarine Core](...).

The exact output format is flexible but the important part is that semantic labels of images are not silently discarded.

[fccview]

This was an easy quick change, I also made it so it can be disabled so if anyone doens't want the image noise (for whatever reason) they can disable it.

I have tested it on a couple of domains and it seems to be working well but let me know before I close (latest rebuild).

Gonna focus on #11 and see if I can quickly append degoog transports in before doing a full on documentation.
Then I'll probably release this to the wild and see if there's more issues coming in!

[quackerd]

I can confirm this partially fixes the issue.

On the dota2 page, now images under ITEM STATS start showing their names but not images in STARTING ITEMS and LATE GAME. For example, the returned result for STARTING ITEMS still doesn't have item names:

scrape result:

Lategame
Most common Inventories in lategame
Minute 20-35
Minute 35-55
Minute 55+

I also attached a screenshot of inspecting the exact relevant element and I hope it helps. I think there was some framing issue in my original issue post. That image does not use an img tag but a div.

[quackerd]

I think this might need more consideration than blindly matching those divs. I am personally not very familiar with HTML so I am just pointing out the "symptoms" and "expected results" here instead of advertising support for that specific div format...

[fccview]

I think this might need more consideration than blindly matching those divs. I am personally not very familiar with HTML so I am just pointing out the "symptoms" and "expected results" here instead of advertising support for that specific div format...

No that's good thank you!
I need to find a way to regex out images, I was hoping targeting any img with cheerio would just work.

I'll spend some time on this tomorrow and do some proper tests on a variety of sites (including yours).
I am using hermes for scrapes and it did say everything was fine but I wonder if it just hallucinated a little, should have spent more time double checking, sorry. I'll keep this open until I can confidently send it back to you, focusing on the transport stuff right now

[quackerd]

Take your time btw the baseline rebuild image now is already pretty workable from the websites/searches I've tested.