-
Notifications
You must be signed in to change notification settings - Fork 90
AwkwardForth documentation
The AwkwardForth language will eventually be documented in a proper place, but since @ianna will need a quick reference while developing TypedArrayBuilder, I'll write something useful here.
AwkwardForth is a subset of standard Forth with some additional built-in words. It is a domain specific language for creating columnar Awkward Arrays from record-oriented data sources, especially for cases in which the deserialization procedure for the record-oriented data is not known until runtime. Typically, this is because the data has a type or schema that is discovered at runtime and that type determines how bytes of input are interpreted and in what order. This does not apply to columnar data sources, such as Apache Arrow, Parquet, or some ROOT data, such as numerical types (like int or float) and jagged arrays of numbers (like std::vector<int>). It does apply to record-oriented sources like ProtoBuf, Avro, and complex types in ROOT TTrees, such as std::vector<std::vector<int>> or unsplit classes. Note that ROOT's new RNTuple is entirely columnar.
The Easy Forth one-page tutorial is an excellent introduction to the idea of Forth. In a nutshell, whereas functional programming strives for pure functions with no side effects, Forth operations consist purely of side effects: every operation changes the state of the machine, whether the global stack of integers, global variables, or in the case of AwkwardForth, positions in input buffers and data written to output buffers. It has almost no syntax, less even than Lisp, in that it consists entirely of whitespace-separated words interpreted in reverse Polish order. (Looping and branching constructs do have a recursive grammar, but they are exceptions.)
AwkwardForth is interpreted as a bytecode-compiled virtual machine. The source code is compiled into sequences of integer codes, one sequence per user-defined word or nested control flow (e.g. body of loops and conditional branches). This "compilation" is literal, like Python or Java bytecode—no optimization is attempted. It is interpreted by a virtual machine that (on my laptop) runs at about 5 ns per instruction. Instructions are 1‒3 bytecodes long, and each bytecode is a 32-bit integer (templated as I in C++, but only instantiated for int32_t). For comparison, the Python virtual machine (on the same laptop) runs at about 900 ns per instruction (see this comment), so AwkwardForth is an "interpreter" in the same sense as CPython, but almost 200× faster, due to its specialization. Strictly mathematical calculations can be much faster in compiled, optimized C++, but strictly I/O operations (from RAM to RAM) is about the same, with C++ being only 1.8× faster in the limit of one 32-bit copy per instruction. If dozens or more bytes are copied per instruction, the gap between AwkwardForth and C++ becomes insignificant. Since AwkwardForth is intended for mostly I/O purposes, this is acceptable.
Forth's emphasis on state-changing operations would make it a terrible choice for vectorized accelerators like GPUs, but an FPGA implementation could be great: FPGAs have a much longer "compilation" time than even C++, so it would be advantageous for an FPGA to be configurable by Forth programs in the same sense as AwkwardForth. Such a thing could, for instance, read ROOT files directly from GHz Ethernet into machine learning models implemented with hls4ml.
This part of the documentation is the most in flux, since we'll likely add features to the ForthMachine to make debugging easier.
In C++, there are three classes:
-
ForthMachineOf<T, I>, where
Tis the stack type (int32_torint64_t) andIis the instruction type (onlyint32_thas been instantiated). -
ForthInputBuffer is an untyped input buffer, which wraps a
std::shared_ptr<void>. (Note that one operation, copying multiple numbers from the input buffer to the stack (not directly to output buffers), will temporarily mutate data in the buffer if they need to be byte-swapped. This is a temporary mutation, so the buffer can be used by other functions afterward, but not at the same time as the ForthMachine. This thread-unsafety could be changed in the future.) -
ForthOutputBufferOf is a typed output buffer, specialized by
OUT. (The fact that the write methods are virtual is not a performance bottleneck: putting the output type information into Forth bytecodes and using aswitchstatement to go to specialized method calls has identical performance for small copies and is up to 2× worse for large copies. C++ vtables are hard to beat.)
In Python, only the two instantiations of the ForthMachine are bound through pybind11:
>>> from awkward.forth import ForthMachine32
>>> from awkward.forth import ForthMachine64The methods available in Python are a subset of the ones in C++. (The fast, lookup-by-integer methods were omitted.)
A ForthMachine compiles its source code once when it is constructed; new code requires a new machine. This machine computes the sum of 3 and 5.
>>> vm = ForthMachine32("3 5 +")
>>> vm.run()
>>> vm.stack
[8]A ForthMachine has 3 states: "not ready," "paused," and "done." There are 6 methods that control execution of a ForthMachine:
-
run(inputs): resets the state of the machine, starting in any state, and runs the main code from the beginning. If control reaches apauseword, the machine goes into the "paused" state. Otherwise, it goes into the "done" state. -
begin(inputs): resets the state of the machine, starting in any state, and goes into a "paused" state before the first instruction in the main code. -
resume(): starts execution from a "paused" state and continues until the end of the main code, resulting in "done," or until the end of a user-defined word, if a word was paused while being called (see below). -
call(word): starting from a "paused" or "done" state, executes a user-defined word. If this operation contains apauseword, the machine will need to be resumed (see above) to reach the end of the user-defined word. When the user-defined word is finished, the state of the machine will be "paused" or "done," depending on where it started. -
step(): executes only one instruction, starting from a "pause" state, ending in a "pause" or "done" state, depending on whether the last instruction in the main code is reached. This only exists for debugging: normal pausing and resuming should be done withpausewords andresume()calls. -
reset(): resets the state of the machine and (unlike all of the above), clears the stack, all variables, and detaches the input and output buffers (which might be significant for cleaning up memory use).
Here are some examples of controlling the execution state of a ForthMachine.
Stepping through a program (for debugging only):
>>> vm = ForthMachine32("3 5 +")
>>> vm.begin()
>>> vm.stack
[]
>>> vm.step()
>>> vm.stack
[3]
>>> vm.step()
>>> vm.stack
[3, 5]
>>> vm.step()
>>> vm.stack
[8]Pausing and resuming execution:
>>> vm = ForthMachine32("1 2 pause 3 4")
>>> vm.run()
>>> vm.stack
[1, 2]
>>> vm.run()
>>> vm.stack
[1, 2]
>>> vm.resume()
>>> vm.stack
[1, 2, 3, 4]Halting execution:
>>> vm = ForthMachine32("1 2 halt 3 4")
>>> vm.run()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: 'user halt' in AwkwardForth runtime: user-defined error or stopping condition
>>> vm.stack
[1, 2]
>>> vm.run(raise_user_halt=False)
'user halt'
>>> vm.stack
[1, 2]
>>> vm.resume()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: 'not ready' in AwkwardForth runtime: call 'begin' before 'step' or 'resume' (note: check 'is_ready')Calling a user-defined word:
>>> vm = ForthMachine32(": callme 1 2 3 4 ;")
>>> vm.call("callme")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: 'not ready' in AwkwardForth runtime: call 'begin' before 'step' or 'resume' (note: check 'is_ready')
>>> vm.run()
>>> vm.stack
[]
>>> vm.call("callme")
>>> vm.stack
[1, 2, 3, 4]Interaction between pause and calling a user-defined word:
>>> vm = ForthMachine32(": callme 123 pause 321 ; 1 2 pause 3 4")
>>> vm.run()
>>> vm.stack
[1, 2]
>>> vm.call("callme")
>>> vm.stack
[1, 2, 123]
>>> vm.resume()
>>> vm.stack
[1, 2, 123, 321]
>>> vm.resume()
>>> vm.stack
[1, 2, 123, 321, 3, 4]Manipulating the stack outside of a program:
>>> vm = ForthMachine32("if 123 else 321 then")
>>> vm.begin()
>>> vm.stack
[]
>>> vm.stack_push(-1) # true
>>> vm.stack
[-1]
>>> vm.resume() # if pops the value and runs the first branch
>>> vm.stack
[123]
>>> vm.begin()
>>> vm.stack
[]
>>> vm.stack_push(0) # false
>>> vm.stack
[0]
>>> vm.resume() # if pops the value and runs the second branch
>>> vm.stack
[321]AwkwardForth can also have (global, scalar) variables, (global, untyped) inputs, and (global, typed) outputs. (The language has no nested scopes.) Here is an example of a ForthMachine with a variable:
>>> vm = ForthMachine32("variable x 10 x !")
>>> vm["x"]
0
>>> vm.run()
>>> vm["x"]
10Here is an example of a ForthMachine with an input (i-> reads data as a 4-byte integer and moves the position 4 bytes):
>>> import numpy as np
>>> vm = ForthMachine32("input x x i-> stack")
>>> vm.run({"x": np.array([3, 2, 1], np.int32)})
>>> vm.stack
[3]
>>> vm.input_position("x")
4Here is an example of a ForthMachine with an output (<- writes data from the stack, converting it to the output type, if necessary):
>>> vm = ForthMachine32("output x int32 999 x <- stack")
>>> vm.begin()
>>> vm.step()
>>> vm.stack
[999]
>>> vm["x"]
<NumpyArray format="i" shape="0" data="" at="0x58c8c85d11c0"/>
>>> vm.step()
>>> vm.stack
[]
>>> vm["x"]
<NumpyArray format="i" shape="1" data="999" at="0x58c8c85d11c0"/>A ForthMachine can have an arbitrary number of variables, inputs, and outputs, and an arbitrary number of user-defined words, with index orders defined by the order of declaration (relevant for fast C++ access).
AwkwardForth has no floating-point operations at all. (If we need to add one, it would be a separate floating-point stack, which is the typical way Forth implementations handle floating-point calculations, if at all.)
The bytecode instructions for an AwkwardForth program are a ListOffsetArray of 32-bit integers, which can be inspected and decompiled.
>>> import awkward as ak
>>> vm = ForthMachine32("if 123 else 321 then")
>>> vm.bytecodes
<ListOffsetArray64>
<offsets><Index64 i="[0 3 5 7]" offset="0" length="4" at="0x58c8c859ef00"/></offsets>
<content><NumpyArray format="i" shape="7" data="4 60 61 0 123 0 321" at="0x58c8c84c9310"/></content>
</ListOffsetArray64>
>>> ak.Array(vm.bytecodes)
<Array [[4, 60, 61], [0, 123], [0, 321]] type='3 * var * int32'>
>>> print(vm.decompiled)
if
123
else
321
thenYou can also get the current position in the bytecode (the position of the next instruction to be run) and a decompiled string of that instruction.
>>> vm = ForthMachine32("1 2 pause 3 4")
>>> # Literal integers in the source code are two-bytecode instructions (0 followed by the number).
>>> ak.Array(vm.bytecodes)
<Array [[0, 1, 0, 2, 2, 0, 3, 0, 4]] type='1 * var * int32'>
>>> vm.current_bytecode_position
-1
>>> vm.begin()
>>> vm.current_bytecode_position
0
>>> vm.current_instruction
'1'
>>> vm.resume()
>>> vm.current_bytecode_position
5
>>> vm.current_instruction
'3'
>>> vm.resume()
>>> vm.current_bytecode_position
-1
>>> vm.current_instruction
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: 'is done' in AwkwardForth runtime: reached the end of the program; call 'begin' to 'step' again (note: check 'is_done')
(https://github.com/scikit-hep/awkward-1.0/blob/1.0.2/src/libawkward/forth/ForthMachine.cpp#L1302)Note that this current_bytecode_position refers to the absolute position in bytecodes.content, not a position relative to the beginning of a segment. The following example illustrates that, as well as the use of current_recursion_depth (PR #653 may be required):
>>> vm = ForthMachine32("0 if 123 else 321 then")
>>> ak.to_list(vm.bytecodes)
[[0, 0, 4, 60, 61], [0, 123], [0, 321]]
>>> vm.begin()
>>> vm.current_bytecode_position, vm.current_recursion_depth, vm.current_instruction
(0, 1, '0')
>>> vm.step()
>>> vm.current_bytecode_position, vm.current_recursion_depth, vm.current_instruction
(2, 1, 'if\n 123\nelse\n 321\nthen')
>>> vm.step()
>>> vm.current_bytecode_position, vm.current_recursion_depth, vm.current_instruction
(4, 1, '(anonymous segment at 2)')
>>> vm.step()
>>> vm.current_bytecode_position, vm.current_recursion_depth, vm.current_instruction
(7, 2, '321')
>>> vm.step()
>>> vm.current_bytecode_position, vm.current_recursion_depth(-1, 1)As the ForthMachine executes code, it counts the number of instructions it encounters and the number of nanoseconds spent in the execution loop. This can be useful for quantifying algorithms.
>>> vm = ForthMachine32("5 3 + 2 *")
>>> vm.count_instructions, vm.count_nanoseconds
(0, 0)
>>> vm.run()
>>> vm.count_instructions, vm.count_nanoseconds
(5, 6739)
>>> vm.run()
>>> vm.count_instructions, vm.count_nanoseconds
(10, 15233)
>>> vm.run()
>>> vm.count_instructions, vm.count_nanoseconds
(15, 23751)
>>> vm.run()
>>> vm.count_instructions, vm.count_nanoseconds
(20, 32512)
>>> vm.count_reset()
>>> vm.count_instructions, vm.count_nanoseconds
(0, 0)In performance studies, keep in mind that only large samples are meaningful, since modern processors streamline code as it runs (moving data/Forth instructions from RAM into CPU cache, predicting branches, pipelining hardware instructions, etc.).
There are also counters for read instructions and write instructions.
>>> vm = ForthMachine32("""
... input x
... output y float64
...
... 10 0 do
... x d-> y
... loop
... """)
>>> vm.run({"x": np.arange(10) * 1.1})
>>> vm["y"]
<NumpyArray format="d" shape="10" data="0 1.1 2.2 3.3 4.4 5.5 6.6 7.7 8.8 9.9" at="0x58cd85e9e340"/>
>>> vm.count_reads, vm.count_writes
(10, 10)
>>> vm.run({"x": np.arange(10) * 1.1})
>>> vm.count_reads, vm.count_writes
(20, 20)Note that multi-read/write instructions (described below) count as one because they are much faster than individual read/writes.
>>> vm = ForthMachine32("""
... input x
... output y float64
...
... 10 x #d-> y
... """)
>>> vm.run({"x": np.arange(10) * 1.1})
>>> vm["y"]
<NumpyArray format="d" shape="10" data="0 1.1 2.2 3.3 4.4 5.5 6.6 7.7 8.8 9.9" at="0x58cd85e9e340"/>
>>> vm.count_reads, vm.count_writes
(1, 1)
>>> vm.run({"x": np.arange(10) * 1.1})
>>> vm.count_reads, vm.count_writes
(2, 2)Also note that the execution reset() is independent of the performance-counter count_reset(). Resetting one does not reset the other.
>>> vm.reset()
>>> vm.count_instructions, vm.count_nanoseconds, vm.count_reads, vm.count_writes
(4, 18769, 2, 2)
>>> vm.count_reset()
>>> vm.count_instructions, vm.count_nanoseconds, vm.count_reads, vm.count_writes
(0, 0, 0, 0)Standard Forth has two types of comments: parentheses and backslash-to-end-of-line.
>>> vm = ForthMachine32("( This does nothing. )")
>>> ak.Array(vm.bytecodes)
<Array [[]] type='1 * var * int32'>
>>> vm = ForthMachine32("1 2 ( comment ) 3 4")
>>> vm.run()
>>> vm.stack
[1, 2, 3, 4]
>>> vm = ForthMachine32("""
... 1 2 \\ comment to end of line
... 3 4 \\ 2 backslashes in Python quotes -> 1 backslash in string
... """)
>>> vm.run()
>>> vm.stack
[1, 2, 3, 4]In both styles, you have to make sure that the "(", ")", and "\" characters are separated by a space; otherwise the tokenizer won't recognize them as distinct from another word. (That is, "(comment)" is not "( comment )".) Also, parentheses are closed by the first balancing close-parenthesis.
>>> vm = ForthMachine32("( outer ( inner ) still a comment )")
>>> ak.Array(vm.bytecodes)
<Array [[]] type='1 * var * int32'>Literal integers in the source code put an integer on the stack. AwkwardForth has no floating point types, so only -?[0-9]+ are allowed, no . or e. If the number is prefixed by 0x, then the number is parsed as hexidecimal, with -?[0-9a-f] allowed.
>>> vm = ForthMachine32("1 2 -3 04 0xff")
>>> vm.run()
>>> vm.stack
[1, 2, -3, 4, 255]User defined words: : .. ;
The main distinction between Forth and a stack-based assembly language is that Forth allows the programmer to define new words. These words are like subroutines, but do not have formal argument lists or return values: they manipulate the stack like any built-in word. A word's "informal" arguments are the items it pops off the stack when it begins and its "informal" return values are the items it pushes onto the stack when it ends.
It is customary to document a word with a comment like
: sum-of-squares ( x y -- sum )
dup * ( x y -- x y*y )
swap ( x y*y -- y*y x )
dup * ( y*y x -- y*y x*x )
+ ( sum )
;That is, the state of the top of the stack (the rightmost end is the "top," where items get pushed and popped) before the operation is to the left of two hyphens "--" and the state of the top of the stack afterward is to the right. Here is that example as a ForthMachine:
>>> vm = ForthMachine32("""
... : sum-of-squares ( x y -- sum )
... dup * ( x y -- x y*y )
... swap ( x y*y -- y*y x )
... dup * ( y*y x -- y*y x*x )
... + ( sum )
... ;
... 3 4 sum-of-squares
... """)
>>> vm.run()
>>> vm.stack
[25]User-defined words are used like any other word—in reverse Polish order. Thus, 3 4 sum-of-squares calls this newly defined word.
In AwkwardForth, words can be defined after they are used, and they can call themselves by name recursively. (Not all Forths allow that.) All declarations (new words, variables, inputs, and outputs) are compiled in a global namespace when a ForthMachine is constructed. However, words can only call previously defined words or themselves because this compilation proceeds in one pass. (It's also possible to define a word inside of a definition of a word, but there is no value in doing so, because namespaces are not scoped and Forth has no notion of a closure.)
Note that a "common error" is to forget a space between the colon (":") and the word it defines or the semicolon (";") and the last word in the definition.
AwkwardForth functions can call themselves for recursion, but the standard defines recurse to allow it in systems without this ability. It is included for convenience in porting examples from other Forths. For example, Fibonacci numbers from this page:
>>> vm = ForthMachine32("""
... : fibonacci ( n -- nth-fibonacci-number )
... dup
... 1 > if
... 1- dup 1- recurse
... swap recurse
... +
... then
... ;
... 20 0 do
... i fibonacci
... loop
... """)
>>> vm.run()
>>> vm.stack
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181]In this example, the word recurse could be replaced with fibonacci.
The if .. then brackets a sequence of words, pops one value of the stack, does nothing if that value is exactly 0, and does the bracketed words if it is non-zero. Conventionally, -1 is used as "true" because it is the bitwise inversion of 0 (in two's complement arithmetic).
Note that the word "then" acts as a terminator of the code branch: it comes after the code to run if the predicate is true. This is just a weird rule to remember.
>>> vm = ForthMachine32("if 1 2 3 4 then")
>>> vm.begin()
>>> vm.stack_push(0)
>>> vm.resume()
>>> vm.stack
[]
>>> vm.begin()
>>> vm.stack_push(-1)
>>> vm.resume()
>>> vm.stack
[1, 2, 3, 4]The if .. else .. then brackets two sequences of words, pops one value off the stack, does the first if that value is non-zero and the second if that value is zero.
>>> vm = ForthMachine32("if 123 else 321 then")
>>> vm.begin()
>>> vm.stack_push(0)
>>> vm.resume()
>>> vm.stack
[321]
>>> vm.begin()
>>> vm.stack_push(-1)
>>> vm.resume()
>>> vm.stack
[123]The do .. loop brackets a sequence of words, pops two values off the stack, "stop" and "start," and repeats the bracketed sequence "stop minus start" times. Note that the top of the stack is the starting value and the second-to-top is the stopping value, so they read backward. Here are two examples:
>>> vm = ForthMachine32("""
... 10 0 do
... 123
... loop
... """)
>>> vm.run()
>>> vm.stack
[123, 123, 123, 123, 123, 123, 123, 123, 123, 123]As described below, i is the current state of the incrementing variable.
>>> vm = ForthMachine32("""
... 10 0 do
... i
... loop
... """)
>>> vm.run()
>>> vm.stack
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]Although the "start" and "stop" values may be constants in the code, they are pulled from the stack, so they can be determined at runtime.
The do .. +loop brackets a sequence of words, pops two values off the stack, "stop" and "start," and repeats the bracketed sequence. At the end of the bracketed sequence, another value is popped off the stack, "step", which indicates how much the incrementing variable changes in each step.
>>> vm = ForthMachine32("""
... 100 0 do
... i
... 10
... +loop
... """)
>>> vm.run()
>>> vm.stack
[0, 10, 20, 30, 40, 50, 60, 70, 80, 90]Like "start" and "stop," the "step" value is pulled from the stack, so it can be determined at runtime.
>>> vm = ForthMachine32("""
... 1000 1 do
... i
... dup 2 *
... +loop
... """)
>>> vm.run()
>>> vm.stack
[1, 3, 9, 27, 81, 243, 729]The letters i, j, and k are reserved words whose values are set by do loops and nested do loops (up to three levels).
>>> vm = ForthMachine32("""
... 10 5 do
... 8 3 do
... 5 0 do
... k 100 * j 10 * i + +
... loop
... loop
... loop
... """)
>>> vm.run()
>>> vm.stack
[530, 531, 532, 533, 534,
540, 541, 542, 543, 544,
550, 551, 552, 553, 554,
560, 561, 562, 563, 564,
570, 571, 572, 573, 574,
630, 631, 632, 633, 634,
640, 641, 642, 643, 644,
650, 651, 652, 653, 654,
660, 661, 662, 663, 664,
670, 671, 672, 673, 674,
730, 731, 732, 733, 734,
740, 741, 742, 743, 744,
750, 751, 752, 753, 754,
760, 761, 762, 763, 764,
770, 771, 772, 773, 774,
830, 831, 832, 833, 834,
840, 841, 842, 843, 844,
850, 851, 852, 853, 854,
860, 861, 862, 863, 864,
870, 871, 872, 873, 874,
930, 931, 932, 933, 934,
940, 941, 942, 943, 944,
950, 951, 952, 953, 954,
960, 961, 962, 963, 964,
970, 971, 972, 973, 974]The begin .. again brackets a sequence of words and repeats them indefinitely. Only an error or a control-flow construct like exit, halt, and pause can break out of it. Programs can be simplified by repeating indefinitely and ignoring errors.
>>> vm = ForthMachine32("input x begin x i-> stack again")
>>> vm.run({"x": np.arange(10, dtype=np.int32)}, raise_read_beyond=False)
'read beyond'
>>> vm.stack
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]The begin .. until brackets a sequence of words and repeats them, popping a value from the stack at the end of the sequence, and using that value to determine whether to continue. If the value is 0, the body repeats; otherwise, it stops. This is a posttest loop: the condition is part of the repeated body.
>>> vm = ForthMachine32("""
... 10
... begin
... dup 1-
... dup 0=
... until
... """)
>>> vm.run()
>>> vm.stack
[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]The begin .. while .. repeat brackets two sequences of words, executes the first unconditionally and, if non-zero, executes the second sequence. At the end of the second sequence, the control returns to the first sequence to re-evaluate the condition. This is a pretest loop: the condition has to be separated from the loop body like the parenthesized condition in a while loop in C:
while (condition) {
body
}The exit word provides a non-local return from a word.
>>> vm = ForthMachine32("""
... : recursive ( n -- n n-1 )
... dup 0= if
... exit
... then
... dup 1-
... recursive
... ;
... 10 recursive
... """)
>>> vm.run()
>>> vm.stack
[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]If you're familiar with other Forths, note that AwkwardForth does not need an unloop to clean up after incomplete do .. loop constructs.
Variables are declared with variable followed by a name.
>>> vm = ForthMachine32("variable x")
>>> vm["x"]
0Variables have the same numerical type as the stack and global scope.
A variable name followed by "!" pops a value from the stack and assigns it to the variable.
A variable name followed by "+!" pops a value from the stack and adds it to the variable.
A variable name followed by "@" pushes the value of the variable to the stack.
>>> vm = ForthMachine32("""
... variable x
... 10 x !
... 5 x +!
... x @
... """)
>>> vm.run()
>>> vm.stack
[15]These are the standard stack manipulation words.
>>> vm = ForthMachine32("1 2 3 4 dup")
>>> vm.run()
>>> vm.stack
[1, 2, 3, 4, 4]>>> vm = ForthMachine32("1 2 3 4 drop")
>>> vm.run()
>>> vm.stack
[1, 2, 3]>>> vm = ForthMachine32("1 2 3 4 swap")
>>> vm.run()
>>> vm.stack
[1, 2, 4, 3]>>> vm = ForthMachine32("1 2 3 4 over")
>>> vm.run()
>>> vm.stack
[1, 2, 3, 4, 3]>>> vm = ForthMachine32("1 2 3 4 rot")
>>> vm.run()
>>> vm.stack
[1, 3, 4, 2]>>> vm = ForthMachine32("1 2 3 4 nip")
>>> vm.run()
>>> vm.stack
[1, 2, 4]>>> vm = ForthMachine32("1 2 3 4 tuck")
>>> vm.run()
>>> vm.stack
[1, 2, 4, 3, 4]Four-function arithmetic. For asymmetric operations (subtraction, division, and modulo), note the order of arguments: second-to-top first, then top.
>>> vm = ForthMachine32("3 5 +")
>>> vm.run()
>>> vm.stack
[8]>>> vm = ForthMachine32("3 5 -")
>>> vm.run()
>>> vm.stack
[-2]>>> vm = ForthMachine32("3 5 *")
>>> vm.run()
>>> vm.stack
[15]Forth, like Python and unlike C and Java, performs floor division, rather than integer division, so negative values round toward minus infinity, rather than rounding toward zero.
>>> vm = ForthMachine32("22 7 /")
>>> vm.run()
>>> vm.stack
[3]
>>> vm = ForthMachine32("-22 7 /")
>>> vm.run()
>>> vm.stack
[-4]Forth, like Python and unlike C and Java, performs modulo, rather than remainder, so negative values round toward minus infinity, rather than rounding toward zero.
>>> vm = ForthMachine32("22 7 mod")
>>> vm.run()
>>> vm.stack
[1]
>>> vm = ForthMachine32("-22 7 mod")
>>> vm.run()
>>> vm.stack
[6]The /mod operation does division and modulo in a single instruction. It pushes two values onto the stack.
>>> vm = ForthMachine32("22 7 /mod")
>>> vm.run()
>>> vm.stack
[1, 3]Division by zero is one of the possible error states for a ForthMachine.
>>> vm = ForthMachine32("22 0 /")
>>> vm.run()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: 'division by zero' in AwkwardForth runtime: tried to divide by zeroUnary functions pop one value from the stack and push the result.
>>> vm = ForthMachine32("12 negate")
>>> vm.run()
>>> vm.stack
[-12]>>> vm = ForthMachine32("12 1+")
>>> vm.run()
>>> vm.stack
[13]>>> vm = ForthMachine32("12 1-")
>>> vm.run()
>>> vm.stack
[11]>>> vm = ForthMachine32("-12 abs")
>>> vm.run()
>>> vm.stack
[12]The min and max words pop two values from the stack and push one.
>>> vm = ForthMachine32("3 5 min")
>>> vm.run()
>>> vm.stack
[3]
>>> vm = ForthMachine32("3 5 max")
>>> vm.run()
>>> vm.stack
[5]Comparison operators pop two values from the stack and either push -1 (true) or 0 (false).
Note that equality is a single "=" and inequality is "<>".
Standard Forth does not have greater-or-equal or less-or-equal, but they are the obvious extensions.
The 0= word checks for equality with zero, which is useful for normalizing booleans to 0 and -1.
Instead of logical operators, Forth has bitwise operators. For invert to serve as logical-not, the non-zero value must be -1, so normalize it with 0=.
>>> vm = ForthMachine32("0 invert")
>>> vm.run()
>>> vm.stack
[-1]
>>> vm = ForthMachine32("-1 invert")
>>> vm.run()
>>> vm.stack
[0]
>>> vm = ForthMachine32("1 invert")
>>> vm.run()
>>> vm.stack
[-2]Likewise, and and or are bitwise-and and bitwise-or.
>>> vm = ForthMachine32("1 2 or")
>>> vm.run()
>>> vm.stack
[3]
>>> vm = ForthMachine32("1 2 and")
>>> vm.run()
>>> vm.stack
[0]Left bitwise-shift and right bitwise-shift are good for bit fiddling.
The false and true words are useful mnemonics for 0 and -1. They make source code easier to read.
AwkwardForth's input and output handling words are not standard Forth, but a reasonable extension of it for this domain-specific purpose.
Input buffers are declared in the same way as variables. If an input has been declared in the source code, it must be provided in the ForthMachine's run(inputs) and begin(inputs) methods.
>>> vm = ForthMachine32("input x")
>>> vm.run()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: AwkwardForth source code defines an input that was not provided: x
(https://github.com/scikit-hep/awkward-1.0/blob/1.0.2/src/libawkward/forth/ForthMachine.cpp#L996)
>>> import numpy as np
>>> vm.run({"x": np.array([1, 2, 3])})All of the words that read from an input buffer have the form: "input-name *-> output-name". The "input-name" is the name of one of the declared input buffers, the "output-name" is either a declared output buffer name or the special word "stack", and the "*->" is a word that ends in "->". There are 46 different words that end in "->". They are described below.
The destination for a read operation can either be an output buffer or the stack. Directly reading from input to output is faster and more information-preserving than reading from input to the stack and then writing from the stack to the output.
Here's an example of reading directly to an output buffer:
>>> vm = ForthMachine32("""
... input x
... output y float64
...
... x d-> y
... x d-> y
... x d-> y
... """)
>>> vm.run({"x": np.array([1.1, 2.2, 3.3])})
>>> vm["y"]
<NumpyArray format="d" shape="3" data="1.1 2.2 3.3" at="0x58cd86525770"/>Here is an example that goes through the stack:
>>> vm = ForthMachine32("""
... input x
... output y float64
...
... x d-> stack y <- stack
... x d-> stack y <- stack
... x d-> stack y <- stack
... """)
>>> vm.run({"x": np.array([1.1, 2.2, 3.3])})
>>> vm["y"]
<NumpyArray format="d" shape="3" data="1 2 3" at="0x58cd8651b720"/>Since the stack of this ForthMachine32 consists of 32-bit integers, the floating-point inputs get truncated before they can be written to the floating-point output.
You'd only want to copy inputs to the stack before copying them to the output if you need to manipulate them in some way, and the only manipulations relevant in parsing are integer operations, such as cumulative sums and identifying seek points.
Reading a batch of data in one instruction is faster than reading the same data in many steps. To read a batch of data, prepend the "*->" word with a number sign (#). This pops a value off the stack to use as the number of items to read.
The following examples result in the same output:
>>> vm = ForthMachine32("""
... input x
... output y float32
...
... 1000000 0 do
... x d-> y
... loop
... """)
>>> vm.run({"x": np.arange(1000000) * 1.1})
>>> vm["y"]
<NumpyArray format="f" shape="1000000" data="0 1.1 2.2 3.3 4.4 ... 1.09999e+06 1.1e+06 1.1e+06 1.1e+06 1.1e+06" at="0x58cd86536de0"/>and
>>> vm = ForthMachine32("""
... input x
... output y float32
...
... 1000000 x #d-> y
... """)
>>> vm.run({"x": np.arange(1000000) * 1.1})
>>> vm["y"]
<NumpyArray format="f" shape="1000000" data="0 1.1 2.2 3.3 4.4 ... 1.09999e+06 1.1e+06 1.1e+06 1.1e+06 1.1e+06" at="0x58cd8691a290"/>but the second is faster because it involves two Forth instructions and one memcpy.
Inputs are untyped; their interpretation depends on the sequence of