Test driver requirements

[jabraham17]

General todo

• improve reports from driver
• solve and remove all todos
• developer documentation #81
• user documentation #82
• convert old test directory into proper tests
• support cloning and building toolchains

features

• Improve test driver to inspect state #3
• Set initial state of standalone function #4

bugs

• every test is currently being run twice
• tests failing when they shouldn't, an issue with how the driver checks for success
• not all generated files getting removed
• allow user to specify simulator build configurations out of tree, to support being able to write tests out of tree better

tests to write

• tests for parser
• tests for ishell functionality
• tests for individual syscalls
• tests for rv32/64 I
• tests for rv32/64 M
• tests for rv32/64 A

Test files

• Source file identification
  • default source files
• expected files
  • subtest expected files
  • default expected files
• compilation args files
• execution args files
• simulator selection
• toolchain selection
• SKIP
• prediff

Arguments

• specify multiple files/dirs to search
• --color
• --parallel
  • partially done
• --dry-run
• --keep
  • partially done
• --exclude
• --exclude-regex
• --diff
• --summary
• --full-report, which prints tests as they finish

features with no planned timeline for implementation

• Reconfigurable test harness #23
• Add perf tests #24