Search: create a more generic solution to extract content

[humitos]

I feel like there must be a more generic solution here. Presumably this problem has been solved a bunch of times, and we can just a library or similar to parse this stuff out?

We're using this on ads, and it's pretty effective: https://trafilatura.readthedocs.io/en/latest/corefunctions.html#extraction

Originally posted by @ericholscher in #12757 (comment)

[stsewd]

I took a quick look, and the only way to get a structured output is using --formatting or --markdown. But we also require extracting the ID to link users to the correct section. So, we would still need to have some custom logic somewhere (if this level of customization is even possible). Haven't searched for more alternatives.