Use TaxCalcBench as a test suite

[ethanc8]

A few months ago, Column Tax (which sells tax preparation software for banks to integrate into their mobile apps) released a benchmark for LLM calculation of US taxes, called TaxCalcBench. The test cases were all created manually, and they intend to update it for future years.

It would probably be useful to integrate TaxCalcBench as a test suite, so that we can ensure our future years' tax calculations are correct, at least for the cases that are included in TaxCalcBench.

The inputs in TaxCalcBench are a proprietary JSON that we would have to figure out how to input to the fact graph. The output is MeF XML; we'd need to find some way to check if two MeF XML files are equivalent.

[elijah-wright]

this says that GPT-5 has 41.67% "strict" correct returns which isn't great. the goal is 100% and it needs to be 100%. I'm looking into consulting for this project and tbh I don't really trust AI

[ethanc8]

Which is exactly why this test suite would be helpful, it gives us a lot of human-prepared examples that LLMs struggle on.

[elijah-wright]

oh so you're saying take the example returns and use those?

[ethanc8]

Exactly, we can figure out how to convert the input data into the fact graph, run the calculations, and compare the output to the sample tax returns.

[elijah-wright]

hm, my priority is getting this working for next year but I can look into this when that's sorted out