Feature to crawl a specific directory across a group of similar websites

[billstout-snc]

As a Vendor Risk Assessment person, I want to collect information from AI Service Providers from their /trust directory to analyze common questions and evidence of certifications. Effectively creating a dataset of trust sites.

[agoyal26]

I would also like to add a requirement for scraping multiple websites (which may not be in same org) but could be having similar content. For example: tutorials on same topic from different learning academies etc.

[obuckner17]

I intend to iteratively expand the feedback loop architecture and continuously assess other opportunities within the AlleyCat LLM.