Skip to content

Validate all paths before doing any expensive work #861

Open
@rohan-varma

Description

We don't currently check all user specified paths for validity (i.e. if directories / checkpoints/ tokenizers actually exist) at the given paths. As a result (for example), we can initialize the model, load in model checkpoint, FSDP shard, etc and then raise an error that the tokenizer is invalid. Instead, we should a-priori check the tokenizer path for validity and save some dev time here.

Metadata

Assignees

No one assigned

    Labels

    better engineeringTasks which help improve eng productivity e.g. building tools, cleaning up code, writing docscommunity help wantedWe would love the community's help completing this issuegood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions