Native test suite for Mathics

[mmatera]

The broader idea is to start doing things the same way if we were using WMA. I don't know in great detail how one writes tests for WMA behavior. However, https://reference.wolfram.com/language/tutorial/UsingTheTestingFramework.html suggests:

test = VerificationTest[2+2, 4]

and test["Outcome"] == "Success".

I have used WMA for 20 years, and I have never written unit tests, or know colleagues that do that. For me, the main reason is that I didn't write WMA code to distribute it, because to distribute, the user should have WMA. I just test my code locally, doing things like

<< mycode.m

Do[
If[(result=Functotest@@parms[[1]])!=parms[[2]], Print["failed",parms, result]Break]
{parms, Transpose[{listofparms,listofvalues}] }]

Is there a simpler commonly used test methodology used in WMA? One could write something simple, such as what is done for gs, but if we could use something that is not custom and current practice, that would be nice. (Nice, but not strictly necessary).

However, having Mathics, it could make sense to define a test suit like pytest to do the tests without making reference to Python. Some of the functions detailed in https://reference.wolfram.com/language/tutorial/ (CreateTest, TestEvaluate) are not even implemented in the CLI of WMA. What WMA supports is a kind of notebook (a test notebook) which has a kind of cells that allow to define tests and see if they pass. I think that what we would like to have is something that runs in the command line.

[rocky]

I have used WMA for 20 years, and I have never written unit tests, or know colleagues that do that.

And the packages I have seen for WMA too often do not come with unit tests. And when this happens, which is too frequent, it is hard to know how to break down a large package into small units that can be tested in isolation so that we can fix bugs in Mathics3.

I have seen this problem in Combinatorica, and KnotTheory. Combinatorica has doctest-like examples from the book that can be run. Generally, there are too few of these cases. Also, the examples contain larger or more complicated situations. What is desired instead are some small, (boring), and simple edge cases that you'd find in a unit test. If a function that exhibits a Mathics3 bug uses other smaller functions, then those generally don't have tests in them.

For me, the main reason is that I didn't write WMA code to distribute it, because to distribute, the user should have WMA. I just test my code locally, doing things like ...

I have encountered this kind of homegrown code a lot. We have it in the gs (Gries Schneider) code. Another variation appears in Greg Chaitin's Unknowable code. Rubi code has its testing regime as well. To me, this does not reflect well on the Programming Language ecosystem.

Some of the functions detailed in https://reference.wolfram.com/language/tutorial/ (CreateTest, TestEvaluate) are not even implemented in the CLI of WMA.

VerificationTest[] works in wolframscript, and that's most of what we need. To the extent we can hook into existing code that would be good. If we have to supply a small amount of code to adapt the notebook-based code to work in a CLI, that's okay for now.

Finally, I should mention that one reason we would like to do this in WMA/Mathics3 code is so that the same tests can be run in both environments to make sure we conform.

[mmatera]

VerificationTest[] works in wolframscript, and that's most of what we need. To the extent we can hook into existing code that would be good. If we have to supply a small amount of code to adapt the notebook-based code to work in a CLI, that's okay for now.

Yep, from the developer's point of view, it is just an alias of Assert[Evaluate[expected]==Evaluate[expr]], not a unit test system.

Finally, I should mention that one reason we would like to do this in WMA/Mathics3 code is so that the same tests can be run in both environments to make sure we conform.

Regarding this and all the other observations I fully agree. My point is that we should develop such a testing system almost from scratch.

[rocky]

Yep, from the developer's point of view, it is just an alias of Assert[Evaluate[expected]==Evaluate[expr]], not a unit test system.

I don't think it is the same. VerificationTest[] is an object that contains a good deal about the test like expected and actual messages, CPUTimeUsed, a test id, the kind of comparison used, and other things. Someone put some thought into this and it possibly reflects some experience in using.

I don't know that we need this initially or if we will need it ever. However, we may get all of this information for free because the code has already been written. And there are likely to be other tools that exist or will be written that make use of the information in a TestResult. And then we will be in a position to use those tools should we need or desire to.

Finally, I should mention that one reason we would like to do this in WMA/Mathics3 code is so that the same tests can be run in both environments to make sure we conform.

My point is that we should develop such a testing system almost from scratch.

I've seen what happens when we develop from scratch. We're not good at this kind of thing compared to whoever wrote VerificationTest. If, for expedience, we need to subset it or rip out parts that don't work, that is okay. Also, in using the existing code, we are finding bugs in our existing implementation such as in the set operators like Union.

[mmatera]

VerificationTest does more or less the same than test.helper.check_evaluation_as_in_cli, so it would be relatively easy to move into a Builtin class.

What I mean with developing from scratch is something equivalent to pytest or WolframWorkbench that takes a wl file, process it, and run the tests. Then we could move all the test_private_doctest_* could be moved to wl files, and tested with this other platform.

[rocky]

VerificationTest does more or less the same than test.helper.check_evaluation_as_in_cli, so it would be relatively easy to move into a Builtin class.

The functionality is almost the same, but there are a couple of big differences. VerificationTest creates a TestObject. The Python code has no such thing. The TestObject has a superset of the information that in the homegrown Python code. The information in TestObject which is returned is available for the user to use and inspect.

Equally important, the code for VerificationTest has already been written! Given the choice of extending our Python code to return a TestObject or using existing already field-tested WMA code that might require us to improve and make more complete our implementation and remove bugs, I think we are better served by improving our code and working on conformance rather than writing custom code that approximates existing WMA code.

As best a I can tell, the System version 11 MUnit is not copyrighted and is/was distributed with Raspberry Pi. So I think we are in the clear for using it.

However having said all of this, I see that WolframWorkbench uses Test, not VerificationTest. Still, everything in principle is the same. This too is a function inside MUnit. It too creates a TestObject.

What I mean with developing from scratch is something equivalent to pytest or WolframWorkbench that takes a wl file, process it, and run the tests.

In trying to understand what WolframWorkbench is, I see in https://reference.wolfram.com/language/tutorial/BuildingLargeSoftwareSystemsInTheWolframLanguage.html#2095156353

One principle says that the last thing to do when building a system is to write any code. You should do everything you can to avoid actual coding. Instead you should focus on your architecture, making tests, and writing documentation. Too many developers take the attitude that they should "dive in and start to code furiously"; often they would do better to stop and think about what they are building.

This is what I think we should try to do with getting MakeBoxes rewritten.

When I look at the "Sample projects" listed in https://www.wolfram.com/workbench/ here is what I see in example-projects/CardDeck/Tests/Test.mt:

(* Mathematica Test File *)

<<CardDeck`

deck = NewDeck[]


Test[
	Length[ deck]
	,
	52
	,
	TestID->"Test-20071129-S5T7T4"
]

Test[
	First[deck]
	,
	Card[{1,1}]
	,
	TestID->"Test-20071129-W2K8B3"
]

We should follow this format in and file organization. I see that CellsToTeX seems to do this as well.