Old questions test cases

[jankrepl]

Closes #693

For reviewers

• There are 59 new test cases
• By default none of the test cases are run. To actually run them you need to set the --exclude-tags=""
• I added the -n / --dry-run flag to the CLI to just enumerate the tests that would be run
• All the test cases I added have the benchmark and $THREAD_ID tags and other tags based on what the user.md is
• Once we merge this PR the idea would be to assign different tags to different people and create new issues

TODO

• Create test cases
  • PLACHOLDER in expected_output.md
  • {} in expected_tool_calls.json
  • "benchmark", $thread_idinparams.json`
• Make sure that test cases that have "benchmark" tag are not run in the CI
• Add dry-run functionality to the CLI