1- FORM Test Suite
2- ===============
1+ # FORM Test Suite
32
43This directory contains a collection of test cases that can be used for
54verifying the behaviour of FORM. It also has a script to run the test cases and
65check the results.
76
8- Prerequisites
9- -------------
7+ ## Prerequisites
108
119The test runner script is written in [ Ruby] ( https://www.ruby-lang.org/ )
12- and requires Ruby 1.9 or later. The script uses the so-called ` test/unit `
10+ and requires Ruby 2.0 or later. The script uses the so-called ` test/unit `
1311library. In some Linux distributions the library is installed together with
1412Ruby, while some distributions may have the library as an optional package,
1513or one may need to manually install
1614[ test-unit] ( http://test-unit.github.io/test-unit/en/ ) via the ` gem ` command.
17- Currently, the script runs only on Unix-like systems.
1815
19- Usage
20- -----
16+ ## Usage
2117
2218### From the build system
2319
2420To use the test suite from the automatic build system
2521(see also the [ INSTALL] ( ../INSTALL ) file),
26- run
22+ run the following command:
2723
28- ```
24+ ``` bash
2925# in the root build directory
3026make check
3127```
3228
3329which tests the executables (release versions) compiled by the build system.
3430
35- ### Testing in the standalone mode
31+ ### Testing in standalone mode
3632
3733Alternatively, one can run the test runner script directly:
3834
39- ```
35+ ``` bash
4036# in the "check" directory
4137./check.rb
4238```
4339
44- By default, it tests ` form ` found in $PATH.
45- To check another executable, give the path as a command line option :
40+ By default, this tests ` form ` found in ` $PATH ` .
41+ To test another executable, specify its path as a command- line argument :
4642
47- ```
43+ ``` bash
4844./check.rb /path/to/form
4945```
5046
51- One can specify a TFORM (or ParFORM) executable in this way.
47+ One can also specify a TFORM (or ParFORM) executable in this way.
5248TFORM and ParFORM will be run with 4 CPUs (can be changed by the ` --cpu N `
5349option).
5450
5551By default, all test cases in all FORM files (` *.frm ` ) found in the ` check `
5652directory (not in subdirectories) are used. To select test cases or FORM files
57- to be run, give their names as command line options, for example,
53+ to be run, specify their names as command- line arguments. For example:
5854
59- ```
60- ./check.rb examples.frm
55+ ``` bash
6156./check.rb Issue8
57+ ./check.rb ' divmod_*'
58+ ./check.rb examples.frm
6259```
6360
64- For more advanced options, see the help message shown by the ` --help ` option.
61+ For more advanced options, refer to the help message using the ` --help ` option.
6562
66- Writing tests
67- -------------
63+ ## Writing tests
6864
6965### Where to add test cases?
7066
71- Currently, the standard test set (run by default) consists of 3 files:
67+ Currently, the standard test set (run by default) consists of 4 files:
7268
73- - ` examples.frm ` : Examples found in the manual.
69+ - ` examples.frm ` : Examples provided in the manual.
7470- ` features.frm ` : Test cases for newly added features.
7571- ` fixes.frm ` : Test cases for bug fixes.
72+ - ` user.frm ` : Test cases contributed by users.
7673
7774Each test case in these files should finish in a short time: the timeout is set
7875to 10 seconds. Bigger tests that take more time are put in subdirectories
79- (e.g., forcer) and should be specified by command-line options when the test
80- suite is invoked.
76+ (e.g., ` forcer ` ) and should be specified by command-line options when the test
77+ suite is invoked:
78+
79+ ``` bash
80+ ./check.rb -C forcer # The Forcer library must be available in FORMPATH.
81+ ```
8182
8283### Structure of a test case
8384
84- A test case is given as a fold in a FORM file. A simple example is:
85+ A test case is given as a fold in a FORM file.
86+ The following is a simple example:
8587
8688```
8789*--#[ Test1 :
@@ -95,16 +97,22 @@ assert result("F") =~ expr("1 + 2*x + x^2")
9597```
9698
9799The fold name ` Test1 ` gives the name of the test case, which should be unique.
98- The part before ` .end ` is a normal FORM program. After ` .end ` , one can write
99- a Ruby program to check the results. In this example, ` assert ` method (which is
100- provided by some unit test class) is used for checking whether its argument is
101- ` true ` . The first assertion checks ` succeeded? ` , which gives ` true ` if the FORM
102- successfully finishes. The second assertion checks the printed result of the
103- expression ` F ` by a regular expression matching (` =~ ` ). In the left-hand side,
104- ` result("F") ` returns the (lastly) printed output for the expression ` F ` as
105- a string. In the right-hand side, ` expr("...") ` makes a regular expression with
106- removing white spaces in its argument. Since ` expr() ` removes all white spaces,
107- one can also put new lines, for example,
100+ The part before ` .end ` is a normal FORM program.
101+ After ` .end ` , one can write a Ruby program to check the results.
102+
103+ The ` assert ` method checks whether its argument evaluates to ` true ` .
104+ In this example:
105+ - The first assertion verifies ` succeeded? ` , which returns ` true ` if the FORM finishes successfully.
106+ - The second assertion checks the printed result of the
107+ expression ` F ` by a regular expression matching (` =~ ` ).
108+ - On the left-hand side, ` result("F") ` returns the (lastly) printed output
109+ for the expression ` F ` as a string.
110+ - On the right-hand side, ` expr("...") ` creates a regular expression
111+ by removing white spaces in its argument.
112+
113+ Since ` expr() ` removes all white spaces,
114+ one can include new lines in the argument.
115+ For example:
108116
109117```
110118*--#[ Test2 :
@@ -123,13 +131,171 @@ assert result("F") =~ expr("
123131
124132which is convenient to copy and paste a long output from a terminal.
125133
126- ### Tips
134+ Two or more FORM programs, separated by ` .end ` , can be put in a test case.
135+ The part after the last ` .end ` is considered as a Ruby program.
136+ For example:
137+ ```
138+ *--#[ Test3 :
139+ S x;
140+ G F = (1+x)^2;
141+ P;
142+ .store
143+ Save out.sav;
144+ .end
145+ Load out.sav;
146+ L G = F;
147+ P;
148+ .end
149+ assert succeeded?
150+ assert result("F") =~ expr("1 + 2*x + x^2")
151+ assert result("G") =~ expr("1 + 2*x + x^2")
152+ *--#] Test3 :
153+ ```
154+
155+ Some test cases need to run only under specific conditions.
156+ In such cases, one can use special instructions starting with ` # ` .
157+ For example:
158+ ```
159+ *--#[ Test4 :
160+ S x;
161+ L F =
162+ #pipe echo "(1+x)^2"
163+ ;
164+ P;
165+ .end
166+ #require unix?
167+ assert succeeded?
168+ assert result("F") =~ expr("1 + 2*x + x^2")
169+ *--#] Test4 :
170+ ```
171+ In this example, ` #require unix? ` ensures that the test runs
172+ only on Unix, where ` #pipe ` is expected to work.
173+
174+ ### Available methods
175+
176+ #### Execution configuration
177+
178+ - ` timeout → integer or float `
179+ Timeout duration in seconds.
180+ - ` ncpu → integer `
181+ Number of assigned CPUs.
182+ - ` total_memory → integer `
183+ Total physical memory available in bytes.
184+ - ` serial? → bool `
185+ ` true ` if FORM is the serial version, otherwise ` false ` .
186+ - ` threaded? → bool `
187+ ` true ` if FORM is the multithreaded version (TFORM), otherwise ` false ` .
188+ - ` mpi? → bool `
189+ ` true ` if FORM is the MPI version (ParFORM), otherwise ` false ` .
190+ - ` valgrind? → bool `
191+ ` true ` if FORM is running under Valgrind, otherwise ` false ` .
192+ - ` wordsize → integer `
193+ Word size in bytes used by FORM (` 4 ` on 64-bit systems).
194+ - ` cygwin? → bool `
195+ ` true ` if running on Cygwin, otherwise ` false ` .
196+ - ` mac? → bool `
197+ ` true ` if running on macOS, otherwise ` false ` .
198+ - ` linux? → bool `
199+ ` true ` if running on Linux, otherwise ` false ` .
200+ - ` unix? → bool `
201+ ` true ` if running on Unix, otherwise ` false ` .
202+ - ` windows? → bool `
203+ ` true ` if running on Windows, otherwise ` false ` .
204+ - ` travis? → bool `
205+ ` true ` if running on Travis CI, otherwise ` false ` .
206+ - ` github? → bool `
207+ ` true ` if running on GitHub Actions, otherwise ` false ` .
208+
209+ #### Job status
210+
211+ - ` return_value → integer `
212+ Exit status of the FORM job.
213+ - ` finished? → bool `
214+ ` true ` if the FORM job finished within the timeout, otherwise ` false ` .
215+ - ` succeeded? → bool `
216+ ` true ` if the FORM job finished without any problems, otherwise ` false ` .
217+ - ` warning? → bool `
218+ ` true ` if the FORM job issued a warning, otherwise ` false ` .
219+ - ` warning?(expected_message : string) → bool `
220+ ` true ` if the FORM job issued the expected warning, otherwise ` false ` .
221+
222+ The following methods are similar to ` warning? ` ,
223+ but they check for preprocessor errors, compile-time errors,
224+ and run-time errors, respectively:
225+
226+ - ` preprocess_error? → bool `
227+ ` preprocess_error?(expected_message : string) → bool `
228+ - ` compile_error? → bool `
229+ ` compile_error?(expected_message : string) → bool `
230+ - ` runtime_error? → bool `
231+ ` runtime_error?(expected_message : string) → bool `
232+
233+ #### Standard streams
234+
235+ - ` stdout → string `
236+ Standard output of the FORM job.
237+ - ` stderr → string `
238+ Standard error of the FORM job.
239+
240+ #### Expressions
241+
242+ The following methods assume the default format for printing expressions:
243+
244+ - ` result(expr_name : string) → string `
245+ The last printed output of the specified expression.
246+ - ` result(expr_name : string, index : integer) → string `
247+ The printed output of the specified expression at the given index (zero-based).
248+ - ` exact_result(expr_name : string) → string `
249+ ` exact_result(expr_name : string, index : integer) → string `
250+ Similar to ` result ` , but returns the exact output, preserving line breaks and whitespaces.
251+
252+ The following methods assume the default format for statistics:
253+
254+ - ` nterms(expr_name : string) → integer `
255+ ` nterms(expr_name : string, index : integer) → integer `
256+ The number of terms as reported in the statistics for the specified expression.
257+ - ` bytesize(expr_name : string) → integer `
258+ ` bytesize(expr_name : string, index : integer) → integer `
259+ The size in bytes as reported in the statistics for the specified expression.
260+
261+ #### Helper methods
262+
263+ - ` exact_pattern(str : string) → regexp `
264+ Regular expression constructed from the given text with escaping any special characters.
265+ - ` pattern(str : string) → regexp `
266+ Similar to ` exact_pattern ` , but ignores whitespaces.
267+ - ` expr(str : string) → regexp `
268+ Similar to ` pattern ` , but matches only with the whole expression.
269+ - ` file(filename : string) → string `
270+ ` read(filename : string) → string `
271+ Text in the specified file.
272+ - ` write(filename : string, text : string) → nil `
273+ Writes a text into the specified file.
274+
275+ ### Available instructions
127276
128- - To verify that FORM finishes with a certain error, one can use
129- ` assert compile_error? ` or ` assert runtime_error? ` .
130- - Two or more FORM programs, separated by ` .end ` , can be put in a test case.
131- Then the part after the last ` .end ` is for Ruby.
132- - To skip a test case for some condition, one can specify it by ` #pend_if ` .
133- (See the result of grepping ` pend_if ` in the existing files.)
134- - When a test case requires other text files, one can use ` #prepare write ` .
135- (See the result of grepping ` prepare ` in the existing files.)
277+ - ` #require <condition> `
278+ Ensures that the test is executed only if the specified ` <condition> ` is met.
279+ - ` #pend_if <condition> `
280+ Marks the test as pending if the specified ` <condition> ` is met.
281+ - ` #prepare <statement> `
282+ Executes the given ` <statement> ` before running the test.
283+ For example:
284+ ```
285+ #prepare write "foo.prc", "#procedure foo\n#message foo\n#endprocedure"
286+ ```
287+ - ` #ulimit <limits> `
288+ Sets the resource limits. This is done via the ` ulimit ` command.
289+ For example:
290+ ```
291+ #require linux?
292+ #ulimit -v 8_000_000
293+ ```
294+ This sets the maximum amount of virtual memory available to
295+ 8,000,000 KiB (~ 8GB).
296+ - ` #time_dilation <dilation> `
297+ Multiplies the timeout by the specified ` <dilation> ` factor.
298+ For example:
299+ ```
300+ #time_dilation 2.0
301+ ```
0 commit comments