We want to create a README.md in the benchmark directory.
Look at the full directory and explore all the code there.
Do NOT look into the "dp_validation_error_analysis" directory. You do not need that.
Explore the other directories in the package.
Make sure you really understand every piece of code in every file in the benchmarks directory and subdirectories (except inside dp_validation_error_analysis) before moving on.
Understand how we run benchmarks via CLI. Then create documentation for how to run benchmarks in a README.md.
Notes
The benchmarks allow users to do a few things:
- manage pipelines and indexes on the deepset platform (creating/deleting) from local yaml configurations -> useful when you want to manually test a setup on the platform
- setup and teardown test cases based on a config -> more formal way of setting up test cases on the platform, useful to add test cases for manual exploration in bulk
- run agents against test cases -> runs agents against one or more test cases and stores evaluation results as well as a full agent transcript (all messages), agent runs perform their own test case setup and tear down
Follow the diataxis documentation framework (roughly):
- getting started
- tutorials
- how-to guides
- concepts
Our documentation is clear and to the point. We make it easily accessible and avoid jargon when it is not needed.
We want to create a README.md in the benchmark directory.
Look at the full directory and explore all the code there.
Do NOT look into the "dp_validation_error_analysis" directory. You do not need that.
Explore the other directories in the package.
Make sure you really understand every piece of code in every file in the benchmarks directory and subdirectories (except inside dp_validation_error_analysis) before moving on.
Understand how we run benchmarks via CLI. Then create documentation for how to run benchmarks in a README.md.
Notes
The benchmarks allow users to do a few things:
Follow the diataxis documentation framework (roughly):
Our documentation is clear and to the point. We make it easily accessible and avoid jargon when it is not needed.