Add SitemapReader originally developed in OERSI - #469
Conversation
Reads sitemap from URL, sends each `loc` URL to the receiver. e.g. `"https://hoou.de/sitemap.xml" | read-sitemap | open-http ...` in a Flux workflow to process every document linked in the sitemap. Supports paging via `from=` query string parameter (see #464) See: https://en.wikipedia.org/wiki/Sitemaps https://gitlab.com/oersi/oersi-etl/-/issues/4 https://gitlab.com/oersi/oersi-etl/-/issues/17
|
Discussed in our planning meeting: we're putting this on hold to investigate if we actually need this kind of specific module for reading sitemaps, or if we can build something based on existing modules and the upcoming paging support (#464). |
|
Kudos, SonarCloud Quality Gate passed!
|
|
@fsteeg i was thinking about this module again. In the last years we saw that pagination mechanism are pretty different. Does it now make sense to move the oersi.sitemapreader to core? |
Do you need the sitemap reader in a non-OERSI context? |
3 participants
Reads sitemap from URL, sends each
locURL to the receiver.e.g.
"https://hoou.de/sitemap.xml" | read-sitemap | open-http ...in a Flux workflow to process every document linked in the sitemap.Supports paging via
from=query string parameter in the sitemap URL.Assigning @dr0i for code review due to the (albeit loose) paging relation to #464.
We don't have a dedicated issue for this, maybe @TobiasNx could do functional review here?