TODO-pilfer.txt

pilfer --no-check-certificates for use with mitm testing etc
DCFandom SiteMap
rss: have a read of https://medium.com/@mandy.michael/building-websites-for-safari-reader-mode-and-other-reading-apps-1562913c86c9
rss: have a read of https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/nav
rss: main element detection should fall back to finding an element with role=main
a debug mode for showing <nav> sections https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/nav
rss - recognise sectioning content: https://developer.mozilla.org/en-US/docs/Web/HTML/Guides/Content_categories#sectioning_content
rss: recognise <main> lements as the main article text? show exclude nav bars etc https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/main
a missing pilferrc should warn, also it fails the UNNAMED_SECTION key lookup
drop all use of URL_re and .url_re, replace with URLPatterns
URLPattern: host and path matches may be any callable returning a mapping?
tyupe safety for zone_keys - prevent eg None for an int etc
SQLTags.preload still seems to do a per entity SQL query
pilfer rss subcommand
SiteMap.updated_entities: can we set start_time to just before the request is dispatched?
common format-shaped pattern to make @on regeps and sitepage formats, custom Formatter class?
should the cache update the X-Cache and X-Cache-Hits headers? MISS/HIT, hit counts
mitm export files to public locations?
some kind of sitemap reload functionality?
a hook after the SNI or CONNECT and before the requestheaders to pass the TLS stream straight through, eg for icloud
SiteMap.patch_soup_top_index to put a toolbar across the top of a page with RSS, dl, etc etc
get the content-length through to stream processing and progressbar
@decorator to preserve the iscoroutineness of the decorated function? And check it works via @OBSOLETE? suggestion by fixerror
cmd_dump: intercept html comments before nav strings
sitemaps: an @on(selector....) instead of the opaque pattern stuff in use presently, flexible generator/return types etc
expose the site-by-sitemap stuff etc as methods on sitemaps?
multiple @listen ? could listen on localhost and fleet2-wg
autoreload modified sitemaps if modified and syntax ok?
pilfer from: can we use the Pilfer cache.find_content over url.GET ?
URLFetcher - if there are cached keys and uncaches keys for a URL, link an old cache entry to the new keys
BUG - if the content comes from the cache the mitm fetch does not happen, so the following stream stages do not run against the content
prefetch should skip+warning URLs with no cache keys
requestheaders should block on in-progress identifcal fetches based on the cache keys?
cache should cache against all sitemap keys
pipe-stage|pipeline runs the stage and pipes its results to pipeline, passing the original input through to the next stage
allow mitm addon classes as actions
√ mitm error and response methods which still call the hook actions
we should not cache incomplete byte ranges?
√ mitm to stream everything if no reponse hook
streaming save action for "pilfer from"
√ divert the mitm logging through upd.print and make into 1 line?
"pilfer from" mirror action?
mitm prefetch action
recognise ~/foo/ in action params as a regexp and compile it?
√ need to merge stream handlers, preserve .content if we have any "response" hooks
the pipelines need to use P.copy_with_vars() for every stage so that vars are inherited and not shared
enclose params in brackets instead of using the : prefix ?
drop Set-Cookie from cached headers along with content-encoding?
graphviz view of pipelines?
pilfer mitm redirector mode for plugging in the adzapper
mitm mode: feed Flow to pipes triggered on Addon.request (full request received) ?
subcommand to dump HTML tags and classes to aid in writing scrapers and recipe decoders etc
shutdown mode: every stage should be inspectable for idleness? wait for all idle?
fill in pipeline staging using function type annotations?
√ use requests for the cs.urlutils URL class
cs.logutils,pilfer: drop D_mode and the D function