Skip to content

Investigate maintaining an index to quickly determine relevant filters #77

@ctz

Description

@ctz

In #76 it was established that a significant amount of time in a check is consumed just reading filters. Most of this work is unnecessary: here is a typical IO pattern for a check that confirms non-revocation:

"20260118-0-default.filter": NotCovered (6337902 bytes)
"20260118-1-default.filter.delta": NotCovered (144518 bytes)
"20260119-0-default.filter.delta": NotCovered (168917 bytes)
"20260119-1-default.filter.delta": NotCovered (160074 bytes)
"20260120-0-default.filter.delta": NotCovered (150195 bytes)
"20260120-1-default.filter.delta": NotCovered (151210 bytes)
"20260121-0-default.filter.delta": NotCovered (131186 bytes)
"20260121-1-default.filter.delta": NotCovered (143546 bytes)
"20260122-0-default.filter.delta": NotCovered (141617 bytes)
"20260122-1-default.filter.delta": NotCovered (136753 bytes)
"20260123-0-default.filter.delta": NotCovered (131369 bytes)
"20260123-1-default.filter.delta": NotCovered (146417 bytes)
"20260124-0-default.filter.delta": NotCovered (148553 bytes)
"20260124-1-default.filter.delta": NotCovered (135185 bytes)
"20260125-0-default.filter.delta": NotCovered (137457 bytes)
"20260125-1-default.filter.delta": NotCovered (136473 bytes)
"20260126-0-default.filter.delta": NotCovered (131625 bytes)
"20260126-1-default.filter.delta": NotCovered (141386 bytes)
"20260127-0-default.filter.delta": NotCovered (181225 bytes)
"20260127-1-default.filter.delta": Good (146882 bytes)

In an ideal world, we could determine more quickly that "20260127-1-default.filter.delta" is relevant to the operation and reach straight for it.

If we maintained an index (computed locally on each fetch) that duplicates the data required for the existing in_universe check in each filter, we could determine (from data in one ~small file) what the relevant filters are.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions