Skip to content

Include example of partial stage reloads in docs #139

Description

@WarmCyan

Curifactory has no way internally of dealing with partial stage cache hits (e.g. one thing in cache for a stage is found but not another)

A way to deal with this manually is through record.stage_cachers:

@cf.stage(None, ["thing1", "thing2"], [PickleCacher]*2)
def make_things(record):
    # in the case where thing1 is in cache but thing2 is not, the whole stage will re-run
    # if we know this is a possibility, and maybe thing1 takes a very long time to compute, we could
    # manually handle this inside the stage (assuming it is not cleaner to separate out thing1 and thing2 production
    # into separate stages)
    if record.stage_cachers[0].check():
        thing1 = record.stage_cachers[0].load()
    else:
        # create thing1 the normal way
        ...
    # create thing2
    ...
    return thing1, thing2

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions