-
Notifications
You must be signed in to change notification settings - Fork 6
Long-Range-Arena Evaluation #49
Copy link
Copy link
Open
Labels
MLRequires machine-learning knowledge (can be built up on the fly)Requires machine-learning knowledge (can be built up on the fly)downstreamChanges code wrapping the core modelChanges code wrapping the core modelengineeringSoftware-engineering problems that don't require ML-ExpertiseSoftware-engineering problems that don't require ML-Expertise
Description
Activity
Metadata
Metadata
Assignees
Labels
MLRequires machine-learning knowledge (can be built up on the fly)Requires machine-learning knowledge (can be built up on the fly)downstreamChanges code wrapping the core modelChanges code wrapping the core modelengineeringSoftware-engineering problems that don't require ML-ExpertiseSoftware-engineering problems that don't require ML-Expertise
Currently, we only know that our model is better than the baseline because of its lower loss at less training time. However, we could run some benchmarks such as LRA to see how well our long-context model performs in a real-world scenario. While LRA doesn't leverage our capabilities ideally (unlike, for example, #5 and #9), it'd still allow us to have preliminary evaluation results on a well-known benchmark dataset.
This issue tacks the progress of integrating our model into LRA, even though it should happen in a separate codebase.