Skip to content

TODO from discussion #4

Open
Open
@rstojnic

Description

@rstojnic

TODO sotabench lib:

  • remove benchmark() function from benchmark.py
  • move deps to requirements
  • evaluation.json should be made if some ENV variable is set, otherwise pprint something
  • for each benchmark:
    • benchmark()
    • default transform
    • the dataset
    • default parameters
  • documentation:
    • dataset examples
    • default transform example
    • input fed to model, and expected output
    • link to examples of benchmarked models
  • a library of transforms (maybe)

And additional requests:

  • BenchmarkResult return value should also contain: 1) the dataset used, 2) the transform used, 3) input parameters used when invoking the function, 4) anything else - so it's a self-contained record of results

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions