Reference Documentation of Java::Geci

Introduction

Java::Geci is a code generation framework/library that makes it very easy to write Java code generating tools. It also

comes with off the shelf code generators that can either replace other widely used code generation tools like the those implemented in the IDEs or Lombok or solve unique tasks that are not available anywhere else, like fluent API generators,
provides services and interfaces to write code generators that need to focus on the very core task of the code generation and let everything else to be done by the framework,
contains very simple generators that can be used as samples and as a starting point to write your own generators. Also, the creation of generators is supported by code generation tools.

The services provided by Java::Geci for the code generators include

Collecting all the source files that are in the source directory.
Filtering the files that need code generation.
Reading unified configuration that the code generator program may need.
Identify the class that was compiled from the source code so that the generators can use reflection to read the structures of the Java application, thus Java source code parsing for the generator is moot.
Provide API for the generator to send the generated code and then the framework will insert the code into the source code.
Extensive API to reflectively query existing classes including an element selector expression that can be used to filter the fields, methods, classes etc. the generator wants to work on.
Java source code generation API to ease source code creation.
Sophisticated and programmable templating engine (Jamal) so that simpler code generators can be implemented templating and without writing a generator specific to the task.

Java::Geci is independent of any other development infrastructure you may or may not use. It is NOT a plugin to any IDE, build tool or testing framework. It is general Java with no special dependency and can be used no matter what type of development tools (IntelliJ, Eclipse, Vim, NetBeans, Maven, Gradle, make, Jenkins, you name it) you use.

Java::Geci comes readily available code genrators, like equals() and hashCode() generator, a delegator and some others. These generators are professional grade and also serve as demo how to create code generators.

Architecture

To understand how to utilize Java::Geci you have to be familiar with the architecture and how it inserts its code generation phase into the build process. The structure is not conventional.

Code generation principally can happen:

(BC) before compilation
(DC) during compilation
(DT) during the test phase <– this is where Java::Geci works
(DCL) during class loading
(DRT) during run-time

In the following we will discuss these different cases. Note that DT, the way Java::Geci works is discussed last because that is the case we will deal with in detail. After all that is how Java::Geci works and this document is mainly all about Java::Geci.

(BC) Before compilation

The conventional phase is before compilation. In that case the code generator reads some configuration, it may read the source code and generates Java code usually into a specific directory separated from the manual source code.

In this case the generated source code is not part of the code that gets into the version control system. Code maintenance has to deal with the code generation and it is hardly an option to omit the code generator from the process and go on maintaining the code manually.

The code generator does not have easy access to the Java code structure. If the generated code has to use, extend or supplement in any way the already existing manual code then it has to analyse the Java source. It can be done line by line or using some parser. In either way this is a task that will be done again by the Java compiler later and also there is a slight chance that the Java compiler and the tool used to parse the code for the code generator may not be 100% compatible.

(DC) during compilation

Java makes it possible to create so called Annotation Processors that are invoked by the compiler. These can generate code during the compilation phase and the compiler will compile the generated classes. That way the code generation is part of the compilation phase.

The code generators running in this phase cannot access the compiled code, but they can access the compiled structure through an API that the Java compiler provides for the annotation processors.

It is possible to generate new classes, but it is not possible to modify existing source code.

(DCL) during class loading

It is also possible to modify the code during the class loading. The programs that do this are called Java Agents. They are not real code generators. They work on the byte code level and modify the already compiled code.

(DRT) during run-time

Some code generators work during run-time. Many of these applications generate java bytecode directly and load the code into the running application. It is also possible to generate Java source code, compile the code and load the resulting bytes into the JVM.

(DT) during the test phase

Java::Geci generates code in the middle of the compilation, deployment, execution life cycle. Java::Geci is started when the unit tests are running during the build phase.

This means that the manual code that was already available is compiled and is available for the code generator and the generator code can access the already compiled code using reflection.

Executing the code during the test phase has another advantage. Any code generation that runs later should generate only code, which is orthogonal to the manual code functionality. It has to be orthogonal in the sense that the generated code should not modify or interference in any way with the existing manually created code that could be discovered by the unit tests. The reason for this is that a code generation happening any later is already after the unit test execution and thus there is no possibility if the generated code effects in any undesired way the behaviour of the code.

Generating code during test has the possibility to test the code as a whole taking the manual as well as the generated code into consideration. Generated code itself should not be tested, per se, but the behaviour of the manual code that the programmers wrote may depend on the generated code and thus the execution of the tests may depend on the generated code.

To ensure that all the tests are OK with the generated code, the compilation and the tests should be executed again in case there was any new code generated. To ensure this the code generation is invoked from a test and the test fails in case new code was generated.

To get this correct the code generation in Java::Geci is usually invoked from a three-line unit test that has the structure:

if( code_was_generated ){
    Assertions.fail("code has changed");
}

The framework decides if the generated code is different or not from the already existing code and writes it back to the source code in the case the code changed but lets the code intact in case the generated the code has not changed.

The method generate() which is the final call in the chain to the code generation returns true if any code was changed and written back to the source code. This will fail the test, but if we run the test again with the already modified sources then the test should run fine.

This structure has some constraints on the generators:

Generators should generate exactly the same code if they are executed on the same source and classes. This is usually not a strong requirement, code generators do not tend to generate random sources. Some code generators may want to insert timestamps as comment in the code: it should not.
The generated code becomes part of the source and they are not compile time artifacts. This is usually the case for all code generators that generate code into already existing classes. Java::Geci can generate separate files but it was designed mainly for inline code generation (hence the name).
The generated code has to be saved to the repository and the manual source along with the generated code has to be in a state that does not need further code generation. This ensures that the CI server in the development can work with the original workflow: fetch - compile - test - commit artifacts to the repo. The code generation was already done on the developer machine and the code generator on the CI only ensures that it was really done (or else the test fails).

Note that the fact that the code is generated on a developer machine does not violate the rule that the build should be machine independent. In case there is any machine dependency then the code generation would result different code on the CI server and thus the build will break.

In the followings, we will describe how to configure and invoke Java::Geci via its API (no external configuration whatsoever is needed, only the API invoked from the tests) and after that how to write code generators.

This documentation is reference documentation. Examples are given in the tutorials listed on the documentation page

Tutorials

Geci invocation API

To use Java::Geci you have to have the libraries on the classpath. If you use Maven then the easiest way is to define the dependencies in the POM file.

<dependency>
    <groupId>com.javax0.geci</groupId>
    <artifactId>javageci-annotation</artifactId>
</dependency>
<dependency>
    <groupId>com.javax0.geci</groupId>
    <artifactId>javageci-api</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.javax0.geci</groupId>
    <artifactId>javageci-core</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.javax0.geci</groupId>
    <artifactId>javageci-engine</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.javax0.geci</groupId>
    <artifactId>javageci-tools</artifactId>
    <scope>test</scope>
</dependency>

Usually it is enough to have a dependency for javageci-core. The other dependencies will be pulled in automatically.

The structure of the invocation is usually one lines in a unit test:

Assertions.assertFalse( ...configuration
                           and invocation of
                           the code generators ...
                        ,
                           "error message that code has changed");

The configuration and invocation of the code generators part of the code is a chained method call that starts with creating a new Geci object and ends with the call to the method generate(). For example the call:

Example:

    @Test
    void testAccessor() throws Exception {
        Geci geci;
        Assertions.assertFalse(
                (geci = new Geci()).source(maven().module("javageci-examples").mainSource())
                        .register(Accessor.builder().build())
                        .generate(),
                geci.failed());
    }

creates the new javax0.geci.engine.Geci object and calls the source() and register() methods to configure the framework. Finally invoking generate() starts the code generation. If there was some new code generated then the call to geci.failed() will return a string detailing which source files were changed.

Geci configuration calls

`source()` defines the alternative directories where the source is

There are several overloaded methods named source() in the Geci interface. They provide means to define different source sets for the generators. Note, that in this context the word "source" has multiple meanings. The generators can read the source files to gather information but at the same time, it can also write generated code to the sources.

The different source() methods usually accept a String array as vararg in the last argument position. This is to specify alternative directories where the source set can be. It is not to specify different source sets. If there are more than one source sets to be used then they should be defined in consecutive source() calls.

The need for this is because there is no guaranteed current working directory (CWD) when the unit tests are started. Many times it is not a problem, but in some cases, for example, when you have a multi module maven project, you may face different CWDs depending on how you start the test. If you start the unit test from the interactive environment of the IDE then the CWD will be the root directory of the module project by default. If you start mvn clean install in the parent project then the CWD is the project root directory of the parent project.

The directory or directories for the source set should be the directory where the Java code hierarchy starts (in case of maven the java directory under src/test or src/main). In other words, it is the directory where the com directory is corresponding to your com. … package structure. In still another words this is the src/main/java or src/test/java directory in Maven terms.

You should not specify a directory deeper in order to limit the source scanning of the framework, because this will prevent finding the class that was created from a certain source file. If there are many packages and sources the generators will (should) ignore them if they do not have anything to do with.

There are two versions of the method source() that has a Predicate as an argument before the alternative directory names. When the directory is selected from the possible alternatives this predicate is used. The leftmost directory in the argument list that tests true with the predicate will be used for the source set. The versions of the method source() that do not have this parameter are using a default predicate that simply checks that the directory exists and it a directory (not a file).

To ease the use of the version that defines predicates there are predefined predicates in the class javax0.geci.api.Source.Predicates. These are:

hasTheFile(String anchor) returns a predicate, which will test true when there is a file named anchor in the directory.
hasOneOfTheFiles(String… anchors) returns a predicate, which will test true when least one of the files named in anchors can be found in the directory.
hasAllTheFiles(String… anchors) returns a predicate, which will test true when all the files named in anchors can be found in the directory.

Named Source Sets

You can also specify a named source set using the source() method. This is needed when the code generator wants to create a new file and there are multiple source sets defined. For example, there can be a source set in the directory src/main/resources containing resource files and another src/main/java containing Java files. If a generator processes a resource file and then wants to generate a Java file then the framework has to create the generated file in the src/main/java directory in the appropriate subdirectory as defined by the package.

Generators that want to create whole new files will specify the source set where they want to create the new file. If there is no named source set in the configuration with the name the generator is seeking then they can not work.

For this purpose the method source() has an overloaded version that accepts the first argument of the type Source.Set. This type is a simple String wrapper to ease readability and help overload. There is a static method in the class Source.Set named set() that can be imported statically and used to specify a source set name. Thus you can specify a named source set in the form source(set("java"),"src/main/java").

Maven directories

Since Maven is the number one build tool and also other build tools use the directory structure standard that was minted by Maven, there is support to specify the source sets in a simple way for Maven projects. There is an overloaded version of the method source() that accept a Source.Maven object as an argument. Using that you can specify the directories in a simple way. You can simply write

source(maven().module("myModule").mainSource())

to specify the main source directory of the module myModule. You can omit the module(…) part of the call in case you have a simple maven project. You can use

mainSource() to have the main source directory as a source set
testSource() to have the test source directory as a source set
mainResources() to have the main resource directory as a source set
testResources() to have the test resource directory as a source set

If you do define none of them, just use source(maven()) or source(maven().module("myModule")) then you define all the four source sets in a single call. This call also defines the set names:

"mainSource" for the main source
"mainResources" for the main resources
"testSource" for the test source
"testResources" for the test resources

It is recommended to use these source set names when you write a generator.

This is the default source set in case you do not specify any source set at all.

Filter source files

You cannot limit the working of the generators to a certain source package or to a certain subdirectory in the source tree specifying the sources but you can filter the actual source files calling the method

only(Pattern …)

When the source directory is scanned for potential source files they are filtered and only those remain in the set of files that match at least one of the patterns specified to the method only() as argument.

When matching the file name against the pattern the UNIX style absolute file name is matched against the pattern. This means that the \ characters are replaced to be / even on Windows operating system.

There is also a form of the method only() that accepts Predicate<Path> values:

only(Predicate<Path> …)

This form gives more freedom to the caller to specify arbitrary selection but the same time it is a bit more complex to use. The regular expressions and predicates work together in the sense that a file will be included into the source set if at least one pattern or predicate lets it get through. The actual implementation of only(Pattern …) converts the regular expression strings on the fly right after they are specified to compiled patterns and then to patterns and when the file filtering is executed the actual code has no sense which predicate was specified as a pattern and which was given by the caller as a predicate.

One call can be used to specify multiple patterns or predicates, but it is also possible to use subsequent calls to only().

Similarly to only() there are two methods, both named ignore(). One has regular expression patterns,

ignore(Pattern …)

the other predicates:

ignore(Predicate<Path> …)

During the collection process a file is only collected if none of the patterns or predicates match the file name. The method ignore() can be used together with the method only().

In addition to this you can exclude whole source sets from source file collections. This makes sense when the source set does not contain any source file that a generator would read and get information from, but the set is named and is used by some generator to create new sources in the set.

To declare that a named source set is for output only you should call output() without any argument chained right after the source() call that specifies the named source. If the source specified in the preceding source() call is not names then the framework throws a GeciException.

As an alternative, you can also call output(Source.Set … sets) to specify which named sets are for output only. This call can be before or after the definition of the source set.

Note that calling any of the filtering methods should be done to increase performance. If the sources should be filtered this way or else you get erroneous output then you probably have a wrong configuration or a generator, which is not configurable flexible enough or simply wrong.

Registering generators

After the source sets are defined, the code generator objects have to be registered so that the framework can call them, one after the other. To do that the method

register(Generator …)

has to be called on the call chain. Generator is the interface that all generators implement. The instantiation of the actual generators is up to the configuration. It is usually just creating the instance using the new MyGenerator() constructor, but in some cases, it may be different if the generator can be instantiated in different ways.

One call can be used to register multiple generator objects, but it is also possible to use subsequent calls to register().

Source Comparator

After the code is generated by all the generators the framework checks if any of the newly generated code differs from those that were already in the source code. This is done simply comparing the individual lines of the original file and the modified code. This will signal modified source even if there is a slight formatting change in the code. The method

comparator(BiPredicate<List<String>, List<String>>)

can be used to change the comparator to something more relaxed. If it is known that the source code is Java code then formatting may be neglected and the predicate may return {@code false} even if the codes are not the same but the difference is only formatting and white space.

Another possible use-case can be when the generator writes some time stamp into the source code to signal the last time the code was generated. In that case a simple though bit more complex than the default trivial source code comparator may mask out the time stamp treating that as irrelevant change and signal only code change if the code was significantly modified.

This comparator should return {@code true} if the source code was changed. The arguments to the bi-predicate are the lists of strings that contain the source code as it was read from the disk (first argument) and as it is after the code generation (second argument).

Tracing the execution

When the code generation does not work the way as expected there is a trace functionality to debug the situation. In some cases the generators do not touch some file, in other cases they may alter some file that they were not supposed to. To get detailed trace information about the actual actions the framework, and the generators do you can specify a file calling the trace("fileName") method on the Geci object before calling generate.

The trace information will ba saved into the trace file in XML format. Although, the XML format is not too sexy, it is practical. If you look at it using some editor that supports XML then you can not only search in it like in case of standard log files, but you can also navigate hierarchically exploding and closing levels of trace information.

View Diff

Java::Geci can be configured calling the diffOutput() method to save the original and the new content into a temporary file. The argument of this function should be a directory name. Java::Geci will save the new content of a file into the folder generated, and the original content into original. These files are generated if there is any difference between the original and the generated content. The generated content may be identical with the original differing only in comment or formatting. In those cases the source file is not modified, and the test does not fail. The diff files are generated even in this case.

The generated files can be opened in any standard diff tool to see what is or what would have been changed.

Generate

The last call after the chain of configuration calls has to be generate(). This call will initiate the code generation and return true if there was some new code generated. This value has to be asserted to be false in the tests and fail in case there was new code generated.

Programming a Generator

A generator is a class that implements the interface javax0.geci.api.Generator. This interface defines a single method

void process(Source source)

The source object represents a single source file and the object should be used by the generator

to read the lines from the actual textual source code if that is needed,
get access to the compiled class that was generated from the given source (such class may not exist in case the source code is not a Java source file),
to get access to writable segments of the source file and to write text into the source
to get access to totally new source files and to write into those generated source code.

Accessing the source in the generator

There are many things that you can reach in your generator code through the source object passed as an argument to the method process() of the generator you write.

The first that comes in front of us is getAbsoluteFile() that will return the absolute file name as a string of the source file that the generator is actually working on. This is rarely needed because the content of the file is available in other ways. It may, however, be needed if the generator limits itself to work only on some specific file (e.g.: it reads only files with the extension .xml), or when the file is binary and cannot be accessed line by line.

If the generator needs to read the text in the file then the method getLines() should be invoked. The return value of the method is a list of String objects that contain the text of the lines. If the generator accesses the content of the file then it does not need to deal with the operating system line ending. The different generators that run in the same process one after the other will also share the content. There is no need to read the content of the file again and again.

It is also important that the source object can be used to modify the content of the file, but only through the declared and guaranteed API. The generator code should not try to replace the lines returned by the method getLines().

Getting segments to write code into

When the generator wants to modify a source file it should use the segments. The source file may contain named segments. These are the lines that are between

<editor-fold id="segment name">

and

</editor-fold>

lines. These lines are edited by the programmer signaling the part of the source code where it expects generated code to be inserted. The generator can access these segments opening a Segment in the source calling the method source.open(id). The parameter is the string that is the specified id="segment name" on the starting line of the segment. Segment object should be used to write lines of generated code. The framework will write back the changes at the end of the execution automatically and also check if the generated code is/was the same as the one that was already in the file. In that case it will not write anything rather it will be happy that code was not changed.

There is a version of the method open() that has no argument. This opens a segment that means the whole source file. A generator should invoke this only when it generates a new source file and the source object was acquired via calling newSource() on the source object. Never invoke the argument-less open() method on the source object that was passed to the generator process() as an argument unless you really know what you are doing. It will delete the content of the source file that was edited by the programmer. Some generators may want to do that but it is their responsibility to write back all the lines into the global segment that was originally in the source file.

Opening a segment is a fairly cheap operation and the generator code can open the same segment many times. The open() method will just return the same segment and the code generation can continue from the place where it was left off after the previous call to open().

Segment initialization

In some cases, code generators happen to generate empty code. In this case code logic may just never call open(id) on a segment, as there is nothing to write there. However, the framework will interpret this that the generator does not want to touch the segment and the old value remains.

Assume that, there is a generator that creates a LC_ prefixed field for every final and static String field which will contain the lowercase version of the original string. There is no such generator written but for the sake of the example, let us imagine one hypothetically. The LC_ prefixed variables created by our hypothetical generator go into a special segment, like

final static String myString = "Hello, World!";

<editor-fold id="lowerCase">
final static String LC_myString = "hello, world!";
</editor-fold>

when the generator does not find any final and static String field it does not have to write anything into the segment named lowerCase and thus it does not open it. This works so long as long there is no final static String field in the class. However, when there are some and we happen to delete them all then this will leave the lowerCase segment intact and there will be the remaining last few LC_… field. That is because the generator does not see any field to be generated and thus it does not try to write into the segment anything. When none of the generators write a segment then the segment will remain as it was.

To tell the framework that the segment is to be modified even if no open(id) is invoked for the specific segment the generator has to call init(id) on the source object. This will essentially delete the already optionally existing content of the segment.

(The call init(id) is essentially the same as open(id). It just happens to be there to emphasize the intent to initialize the segment.)

Lexical Modification

Some generators may want to modify the part of the source code that is manually maintained. Like fixing something in the program. For example inserting the final keyword in front of a field declaration. Or, perhaps deleting some modifier. Such modifications are rare and should be programmed in the generators with great care. The generators really should know what they do as they play with a part of the code that is manually maintained and not separated. Such modifications usually focus on some specific part of the code. To work with a specific part of the code, the application has to find the part. To find the specific part of the code, for example the declaration of a given field is not simple reading the source as text. One can try to apply regular expression searched to the source lines, but that is not reliable. What is really needed to be done is to do a lexical analysis of the source text. This is what Java::Geci can do for the generators.

Overview

Lexical Modifications work with the list of the lexical elements of the source files. A typical modification searches a certain part looking for some specific pattern and then insert or replace the found pattern with a given list of lexical elements.

An example is the record generator. The record generator mimics the behaviour of a Java record, which is not available by the time of writing. To mimic a Java record a class

should have only final fields,
the class itself has to be final,
it should not extend any other class (except the implicit Object) and
it should have a constructor that initializes the final fields and
it has getters for these fields.

The generator reads the fields and creates the constructor as well as the getters. When the programmer starts to create the source code of the class it can insert the field declarations, but they cannot be denoted as final because at this point the constructor initializing them does not exist. The constructor will be created only later to initialize the new field or fields. After the code generation the manual code can be altered so that the field becomes final.

To ease the life of the programmer the Record generator inserts the final keywords where they are needed during the code generation when it also generates/modifies the constructor.

To do that the generator creates a JavaLexed object using the source object. This JavaLexed object contains the java source code lexical elements and provides methods to search and modify the list of the lexical elements. At the creation of the object it "borrows" the source code from the source object and when it finishes and the closes (a JavaLexed object is AutoCloseable) then it returns the source code to the source object. If the list of the lexical elements was changed then the change will be reflected in the source strings.

A generator should never modify the source writing into segments while a JavaLexed object is open.

The recommended pattern to use the JavaLexed object is to use it in a try-with-resources block:

try (final var javaLexed = new JavaLexed(source)) {
    ... analyze the lexical element list and possibly modify it ...
        }

This is the structure you can see in the Record generator. We will use that generator as a sample for the explanation how to use this class. The generator defines a selector expression

private static final Selector<Class<?>> NOT_FINAL = Selector.compile("!final");

Using this selector expression the generator decides whether the class it modifies is final or not:

private void makeClassFinal(Class<?> klass, JavaLexed javaLexed) {
    if (NOT_FINAL.match(klass)) {

This if statement is executed when the class is not final. The generator, in this case, modifies the class to be final, because that is a requirement by the JEP defining the "Record" functionality for Java. In case the developer forgot to make the class final this can be detected and this can also be changed.

When the class is not final the javaLexed object is used to find the declaration of the class in the list of the lexical elements. This is used utilizing the find() and fromStart() methods of the JavaLexed class. This will find the lexemes class followed by the name of the class in the list of the lexemes. The finding process ignores the spaces between the lexemes, because they usually are not important when trying to find some keywords.

When the find is successful the replaceWith() method is executed and this replaces the

"class (spaces) class name"

lexeme section in the list with the specified "final class class name" sequence.

javaLexed.find(list("class", klass.getSimpleName())).fromStart()
               .replaceWith(Lex.of("final class " + klass.getSimpleName()));

As we mentioned earlier, spaces are not important in the search expression. That is because when trying to find a sequence of lexical elements the matching algorithm ignores the spaces and the comments that may be in the source code as not important.

However, when we insert lexical elements as a replacement for some part of the found lexical elements sequence the lexical elements are inserted as they are present and the replacement process will not insert separating white spaces for us. The code can ignore spaces when they come on the input, but they do not know where to put them into the output. Therefore in the replacement lexical element list it is important to have spaces at the appropriate locations.

For example the string

"final class " + klass.getSimpleName()

is converted to a list of lexical elements by the method Lex.of(). The lexical elements will be

keyword final
space
keyword class
space
identifier that is the simple name of the class, e.g. MyClass.

It is important to have a trailing space after the keyword class in the string otherwise the resulting list of lexical elements would not have that space lexical element between the keyword class and the name of the class. That would result final classMyClass instead of final class MyClass when the modified list of lexical expression is given back to the Source object and the list of source lines are recreated.

When the try-with-resources block finishes the modified list of lexical elements is converted back to the list of source line strings that is stored in the source object and the generator can go on and continue to generate code that will get into the segments.

The lexical elements that the generator can work with are objects that are javax0.geci.javacomparator.LexicalElement. The javaLexed object can provide an iterator that goes through the elements returned by lexicalElements() , and it can also provide a specific element based on its index when calling get(i) on the object. It is also possible to remove a specific element calling remove(i) using the index i and there is a method to remove a range of lexical elements calling removeRange(i,j). (Note that the element with the index i is removed but the element with the index j is not.)

Simple Modification Methods

There are also simple and primitive methods to modify the list of the lexical elements. You can add(i, element) to add a single element at the position i. You can also replace a range of elements with a list of lexical elements calling replace(start, end, elements).

These are low level and simple modifications that are available as a last resort in case the available higher level methods do not suffice. The usual way to modify the list of the lexical elements is to use the search and then to replace the found parts with something composed of new lexical elements and elements taken from the found part.

The example above was a very simple one: it was searching for a simple keywords and identifiers as they come one after the other. The search method find() provides much more powerful possibilities. The possibilities are like searching a sub-string literal versus trying to match a regular expression in a string.

Finding and Replacing LEXPRESSIONS

The argument to the method find() is a Lexpression (lexical expression), which is similar to a regular expression when we process strings. In this case, however, instead of characters we build up the expression from lexical elements. Also there is no string representation of the Lexpression like regular expressions have. The way to create a search Lexpression is to use a fluent API provided for the purpose.

The technical implementation of the expression is a BiFunction but it should not matter to the programmer using the API. This is an internal detail.

The way to create such an expression is to use the utility methods provided in the class LexpressionBuilder.

Tip

The utility methods are generated automatically based on the methods of the class Lexpression. For this reason the methods that the generator is calling directly do not have JavaDoc documentation. However, each and every of such method has a method with the same name in the class Lexpression, which is fully documented.

The simplest Lexpression matches a single keyword. To get such a Lexpression you can invoke the method LexpressionBuilder.keyword(kw) with the string argument kw. For example keyword("int") will return an Lexpression that matches a single keyword int.

Note

int is not a keyword in Java. When matching a list of lexical elements against a Lexpression the analysis does not make difference between an identifier and an expression. Because of this the method keyword() and identifier() are interchangeable. It is recommended to use keyword() for keyword matching and identifier() to match an identifier.

Note	In this section we will refer to the methods using their name without the class name. Since the `Lexpression` describing structures can sometimes be fairly complex the recommended approach is to statically import these methods from the class `javax0.geci.lexeger.LexpressionBuilder`.

The method keyword() is not the only one method that results a single, one element Lexpression with a so called terminal symbol in it. There are different methods, like

string() matching a string literal
character() matching a character literal
comment() matching a comment
floatNumber() matching a floating point literal
integerNumber() matching an integer literal
number() matching any number literal (floating point or integer)
match(string) matching a lexical element that is specified a string
type() matching a type definition, like List<String>

Note

the last method, type() matches something that is not a single lexical element, and thus it is not a terminal symbol. A type literal in Java is a complex thing and cannot be described with regular expression structure. To overcome this and to provide a matching possibility for the generators there is a small syntax analyzer implemented in the class TypeMatcher that matches a type literal. In the generator code this method can be used as if a type literal was really a terminal symbol.

Each of these methods have different forms with different arguments. For example the method integerNumber() has a version that has a long predicate as argument. The resulting Lexpression will match an integer number literal only the number converted to long matches the predicate. Similarly the method string() has a version that has a String and one that has a Pattern argument. The first version matches a string lexical element if the value is the given string. The second version matches the regular expression pattern against the actual string value of the lexical element.

All of these methods and many other methods have a version where the first argument is a GroupNameWrapper. When a method xxx() has this version then calling xxx(group(name), …) is the same as calling goup(name, xxx(…)) with the same … arguments.

A GroupNameWrapper can be created calling the method group(name) that wraps the string name into a GroupNameWrapper object. The wrapping helps to distinguish the overloaded methods that have already first string arguments and the same other arguments.

Groups are identified by the name and they contain the list of lexical elements that are matched by the certain sub Lexpression that may be part of the whole Lexpression. These groups can be retrieved using the name and used in the replacement part later.

Matching single elements is not really useful. To have really useful things the builder provides building blocks to compose complex structures recursively using the simple methods. The builder class defines different versions of the following methods:

anyTill() matches zero or more lexical elements until it finds one matching one of the arguments
list() matches the `Lexpression`s listed as arguments
not() matches something that is not the Lexpression argument
oneOf() matches one of the arguments
zeroOrMore() matches zero or more times the argument
oneOrMore() matches one or more time the argument
optional() matches nothing or the argument if possible
repeat() matches the argument many times, min and max values can be specified
unordered() matches the arguments, each once but in arbitrary order

These methods also have different versions overloading the method name.

Examples

In this section we will list some of the test methods that are implemented in the class javax0.geci.lexeger.TestMatching in the module javageci-tools. We list here only some of the methods that are of exemplary values. In the class you can find more examples without detailed explanation.

In the tests the class TestSource is an implementation of the interface Source that gets the lines as the constructor argument. In these tests usually there are only one line in the source code and thus the argument is created calling the static method Collections.singletonList().

The JavaLexed object is created from the source object in a try-with-resources block, essentially implicitly closing it at the end of the test.

   @Test
   void testSimpleListFinding() {
1.     final var source = new TestSource("private final int z = 13;\npublic var h = \"kkk\"");
2.     try (final var javaLexed = new JavaLexed(source)) {
3.         final MatchResult result = javaLexed.find(match("public var h")).fromStart().result();
4.         Assertions.assertTrue(result.matches);
5.         Assertions.assertEquals(13, result.start);
           Assertions.assertEquals(18, result.end);
       }
   }

This test uses the source code as seen on the line #1. The variable result gets the result of the matching operation. The call chain first specifies the Lexpression to find then it executes the operation starting at the beginning of the source and finally the result object is fetched from the result of the search operation.

What we find here is a list of lexical tokens: public, var, and h. This is simply written as match("public var h") which performs a quick lexical analysis on the input string and creates an Lexpression that is the list of the terminal elements that are in the string.

The result.matches is true when the Lexpression was found. The result can also be queried for the start position (inclusive) and the end position (exclusive) of the found match. In this case the lexical elements are

private
space
final
space
int
space
z
space
=
space
13
;
new-line
public
space
var
space
h
space
=
space
"kkk"

Note that spaces do not matter when matching, but it does not mean that they are not there. It is only the comparison that ignores the spaces. In this case the Lexpression given as "public var h" starts at the position 13 in the list of the lexical elements and ends before the elements at the position 18. These positions can be, but need not be used later in replacement operations.

There are different methods that replace the part of the lexical list of the source code between two indices or between the star and end indices of a MatchResult. The simplest way is, however, calling the method replaceWith() that has only one argument, the list of the lexical elements that will be inserted into the place of the matched part only in case the match was successful.

The next sample stores part of the match in a group.

01. @Test
02. void testSimpleGroupCollection() {
03.     final var source = new TestSource("private final int z = 13;\npublic var h = \"kkk\"");
04.     try (final var javaLexed = new JavaLexed(source)) {
05.         final var result = javaLexed.find(list(oneOf(group("protection"), "public", "private"), match("var h"))).fromStart().result();
06.         Assertions.assertTrue(result.matches);
07.         Assertions.assertEquals(13, result.start);
08.         Assertions.assertEquals(18, result.end);
09.         Assertions.assertEquals(1, javaLexed.group("protection").size());
10.         Assertions.assertEquals("public", javaLexed.group("protection").get(0).getLexeme());
11.     }
12. }

What we are searching is a list of Lexpressions. The list consists two elements. The first one is a oneOf() public and private element, the second is var h. The list is created calling the list() utility method that has variable number of Lexpression arguments. The method oneOf() has many overloaded versions. This version has a first argument that specifies a group by name. The rest of the arguments are strings. This is a convenience method that converts the individual strings into simple Lexpressions each. Thus

oneOf(group("protection"), "public", "private")

is the same as

oneOf(group("protection"), match("public"), match("private"))

When this part of the expression matches then the list of lexical elements that are matched by this part of the expression is stored in a new list and is associated with the name and can later be retrieved from the JavaLexed object. The assertions on the line 9 and 10 are doing exactly this. They retrieve the list of the lexical elements that were associated with the group named protection. Line 9 asserts that the number of lexical elements in the list is one and the line 10 asserts that this is public as it is matched before the var h part of the source.

The following example uses a matching group that is not matched during the search operation. Because the structure of the test is practically the same as that of the previous we skipped some lines that you can also see from the line numbering.

1. javaLexed.find(list(group("protection", oneOf(match("public"), group("private", match("private")))), match("var h")));
7. Assertions.assertEquals(0, javaLexed.group("private").size());
8. Assertions.assertEquals(1, javaLexed.group("protection").size());

In this case the method oneOf() has Lexpressions. You cannot mix strings with Lexpressions and because the second argument group("private", match("private")) is already a Lexpression the first one also has to be. Thus we cannot write there simply "public" instead of match("public").

In this example we also use the version of the method group(), which is not inside but around the the matching elements that are to form the group. Using group("protection", oneOf(…)) is the same as the version of oneOf(group("protection",…)) that has a group identifier in the first argument (created calling the single string argument of the utility method group()).

Because the access modifier in front of var h is public the group named private does not match anything thus the size of it is zero. On the other hand the group named protection contains one element. This is the same as in the previous example. If the access modifier was private then both groups would contain a single element list.

Creating, opening new source object

The generator can call newSource(fileName) on a source object. This will create a new source object that can be used to read the content of the named source file if it exists or to write generated code to it and the code will be written into the file named in the argument at the end of the execution unless the code was already there and did not change.

Since the new source code, most of the time is generated in the same directory where the other source code is, the fileName is relative to the file name of the source the newSource() was invoked on.

There is also a version of the method that accepts two arguments: newSource(Source.Set set, String fileName). This can be used when the generated code has to be in a different source set than the one containing the information the generator reads. Even in this case, the directories will be relative to the source just in the different source set. For example, there is the file com/javax0/javageci/Bean.xml in the source set starting in directory src/main/resources/. It contains some description of the bean the generator has to generate. There is another source set defined with the name "mainSource" in the directory src/main/java. Calling source.newSource(set("mainSource"),"Bean.java") will create the file src/main/java/com/javax0/javageci/Bean.java.

Note that generators are encouraged to use Geci.MAIN_SOURCE, Geci.MAIN_RESOURCES, Geci.TEST_SOURCE and Geci.TEST_RESOURCES string constants defined in the interface javax0.geci.api.Geci instead of the string literals.

The code will only be generated only if the global or a named segment was initialized, opened during code generation. If the source was only used to read information and no segment was opened then the file will not be touched by the framework.

When the new content is written back to the file the directories along with all needed parent directories are automatically created.

Accessing the class of the source

Most of the time the source object refers to a Java source file. Since the code runs during unit test execution the compiled version of the class is available and can be examined by the generator using reflection. The name of the class and the package can be deducted from the file name. The suggested way to do this is to invoke the methods provided by the source object for the purpose.

getKlass() returns the class that was created by the source during the compilation phase. If there is no such class then the return value is null.
getKlassName() returns name name of the class. This includes the full package name dot separated.
getKlassSimpleName() returns the simple name of the class file.
getPackageName() returns the name of the package.

There are support methods in the tool module that help with reflection. Before starting to write your code from scratch consult those methods. They contain significant experience.

For example when a generator wants to generate code for each field or each method then this is vital that the order of the fields or methods is the same on different Java versions. There may be different Java build on the developer machine and on the CI server and the reflection method getDeclaredFields() may return the fields in a different order. This causes code generated different on the CI server from the one generated by the developer and thus the CI build fails with unit test error. (It really happened.) To avoid that there are methods that collect fields, methods etc in a sorted definite order in the tool module.

Writing into a segment

After you get access to a Segment object you can use that object to write into the source code into the segment. Whatever you write into a segment will replace the old content. Opening a segment many times, however, does not overwrite the content that the generator was already writing into the segment. For example, a generator creates a setter and a getter for each field in the class. As the generator iterates through the ordered list of the declared fields it opens the segment named "setters" for each field. The generated code will be appended each time and finally replacing the content that was in the file before the code generation.

To write into a segment there are four methods:

write(…) write a line into the code.
write_l(…) write a line into the code and then set the tabstop indented.
write_r(…) unindent the tab stop and then write a line into the code.
newline(…) insert an empty line.

The write…() methods accept a String format and a variable number of objects as parameters. The format string will be used in the String.format() method. Please read the Java documentation on how to use the formatting.

It is also possible to define parameters for a segment. The method param() accepts string pairs (even number of strings) each pair being a key and a value. The parameters defined this way can be used in the strings passed to write(), write_r() or write_l() so that every {{key}} will be replaced by value. Note that the only requirement is that key and value are strings. It is not necessary that key is an identifier, though in the usual cases it helps readability. If a key for a {{key}} is not defined it remains untouched.

You can also reset the parameters calling the method resetParams() on the segment object.

When the line itself contains newline characters then the indenting will automatically be kept for each line. There is no need to spit up the generated multi-line string into lines and invoke the write() method several times. You can write multi-line code safely well tabulated using these methods. This feature can be neatly used with Java 12 multi-line strings.

There are two methods _l() and _r() that are just aliases to write_r() and write_l(). Their use can increase readability when the calls to write()… methods are chained. On the other hand they look ugly when used on their own. Please use them with consideration.

The generator can get access to a temporary segment calling source.temporary(), that does not belong to any source code but is able to collect generated code via the write_X() methods. The code in such or any other segment can be appended to the code of a different segment calling the method write(Segment). (There are no write_r and write_l variants of this method.)

The Segment also has a close() method, that actually does nothing, but Segment also implements AutoCloseable thus it can be used in try-with-resource blocks. It may improve code readability.

Note that the methods provided by the implementation of Segment are simple and they do not want to be a full-blown code generation tool. There is an experimental class javax0.geci.tools.JavaSource that provides more possibilities to generate Java source code. It was mainly used to create the fluent API code generation and also the class API itself is generated by itself demonstrating recursive iterative code generation development. If even the functions provided there are not enough you can use any external library together with Java::Geci.

Generator Parameters

Generators are free to use any configuration they like, however, there are supported configuration ways. Generators can be configured on the application, instance and source level. There is support for the source level configuration. For higher level configuration the tools provided by the Java language and infrastructure is sufficient.

Application level configuration can be hard coded into the class as parameter or can be read from properties files or from other sources. There is no special support for this in Java::Geci. You should follow the usual Java conventions in your code. These parameters affect the behavior of the application for all the runs in the JVM.
Instance level configuration can be done via constructor parameters or via setter or other configuration methods. This is, again, standard Java practice, nothing specific to Java::Geci. These parameters affect the behavior of the application for the instance they were provided.
Generators read source level configuration from the source. These parameters influence the behaviour of the generator when it is processing the specific source. Although generators read the source and could get parameters from many structures, there is support to get the configuration from annotations or from comments. The rest of the section is about the supporting tools that help the generators to read these configuration parameters.

CompoundParams objects

The generators, which work on a specific Java class can access a CompoundParams object using the

CompoundParams global = Tools.getParameters(xxx, mnemonic());

calls. In this call the parameter xxx can be the class, a method or field that the code generator works with and which element is annotated. The second argument is the name/mnemonic of the code generator. The method will return parameters only from the annotation that control this code generator using this value in case there are multiple @Geci annotations on the element xxx for different code generators.

Usually there is an annotation on the class itself and also on the fields or methods. In that case the method can be called on the Class object and also on the Field or Method objects.

When the different configuration parameters are defined on both the class level and also on the field or method level then the code generator usually wants to use the lower level configuration if it exists and the Class level only when the parameter is not defined on the Field or Method level. To ease that configuration handling there is a constructor of CompoundParams that accepts two other CompoundParams as parameters. For example the call

var params = new CompoundParams(local, global);

will result a CompoundParams object that will return the configuration value for any configuration key from the global parameters only if the key is not defined in the local level.

The method Tools.getParameters(xxx, mnemonic()) collects the configuration parameters from the annotation @Geci, which is on the xxx element. There is an annotation interface ready to use defined in the library com.javax0.geci:javageci-annotation but the actual code can use any annotation that is a Geci annotation.

The definition of the Geci annotation is recursive. A Geci annotation is an annotation that

is named Geci
is named in any way but the annotation itself is annotated with a Geci annotation

and

defines at least the value
the type of value is java.langString or java.langString[].

Using your own annotation may eliminate the need for the dependency on the library com.javax0.geci:javageci-annotation.

In case the type of the annotation is String array then the code may use multiple strings as value. These strings will be concatenated with a single space between them. This allow the code to break the annotation into multiple lines in case there are many parameters.

The value of the annotation has to be a string that has the format:

mnemonic option1='value' .... optionN='value'

The options are, well, optional. The value of the options have to be enclosed between apostrophes. If you use a custom annotation that has other parameters in addition to value then those parameters that have String value will also be considered as options for the code generators and they will get into the CompoundParams object.

When you implement a generator extending the class AbstractGenerator then you get the Class object as well as the global configuration as a parameter.

It is also possible to get configuration from the source code without using reflection. The generator may call the static method

CompoundParams getParameters(Source source, String generatorMnemonic, String prefix,
                             String postfix, Pattern nextLine);

This call will scan the source code and try to find the configuration string in the source code, typically placed in some comments. This configuration can be used in case the generator is working from some source file, which is not Java source code and thus there is no corresponding Java class during the test execution. A generator may also use this call in case the application does not want any @Geci annotation to be part of the production code. The drawback of this configuration is that the configuration can only be on the source level and can not be on the Field, Method or other class member level.

Special Generators

When you write a generator you do not need to manually implement the interface javax0.geci.api.Generator. The library contains abstract classes that implement the interface and do some specific task that may be the same for a variety of generators.

These abstract classes are defined in the package javax0.geci.tools in the module javageci-tools. This documentation lists some of the classes giving some introduction, but it is still recommended that you have to consult the actual and up-to-date JavaDoc documentation.

AbstractGeneratorEx can be extended by generators that may throw exception. Note that the signature of the method process() in the interface Generator does not throw any exceptions. This abstract class catches any exception and rethrows it as a RuntimeException.
AbstractGenerator is to be extended by generators that work only on Java source files and need the compiled class of the source.
AbstractDeclaredFieldsGenerator is for generators that want to generate code for each declared field in a class.

AbstractGeneratorEx

The interface Generator defines the method process() in a way that it should not throw exception. If there is an exception during code generation then it has to wrapped into some run-time exception. This will be propagated to the unit test level and thus the test will fail, as it should.

For the wrapping Java::Geci provides the exception class javax0.geci.api.GeciException.

AbstractGeneratorEx implements the method process() invoking the abstract method processEx() it defines wrapping the call into a try-catch block. If there is any exception thrown from processEx() then it is wrapped into a GeciException and thrown.

The abstract method generators must implement in this case is

public abstract void processEx(Source source) throws Exception;

Note that the method may throw Exception and the implemented process() catches only Exception and not any Throwable.

AbstractGenerator

This abstract generator implements the method processEx() and calculates the class of the source and also collects the parameters defined in a @Geci annotation. Extending classes should implement the abstract method

public abstract void process(Source source, Class<?> klass, CompoundParams global)throws Exception;

This case the method is named process() as it has different arguments than the one in the interface and is an overloading of the interface method. The arguments are the

source is the source object
klass is the class of the source
global contains the parameters that are defined in the @Geci annotation on the class level.

AbstractDeclaredFieldsGenerator

This abstract generator does everything as AbstractGenerator essentially extending that class and iterates through the fields of the class. It defines one abstract method that generators extending this class have to implement:

public abstract void processField(Source source, Class<?> klass, CompoundParams params, Field field) throws Exception;

This method is invoked for every field. The parameter params is the composition of the parameters defined on the class level and on the field in @Geci annotations. If a parameter is defined on the field then it prevails, otherwise the one on the class is used.

The class also defines two do-nothing methods that can optionally be overridden by the extending class. These are:

public void preprocess(Source source, Class<?> klass, CompoundParams global) throws Exception {
    }

public void postprocess(Source source, Class<?> klass, CompoundParams global) throws Exception {
    }

As the name suggest preprocess() is invoked before the fields iteration starts and postprocess() is invoked after that.

Error messages

The generators did not touch any source

When the execution finishes and none of the generators generated any source into any existing or new source file then this error happens. If any of the generators generated something but it is the same what was already there then this error does not happen. This error happens if the generators think they have nothing to do. It is usually a configuration error, because if none of the generators generate anything then why to have them in the actual project. They are configured to run, most probably they are supposed to do something.

SourceSet "…" does not exist

When you configure the sources using the source() method in the test executing the generations you can define different source sets and you can also name the sets. When a generator creates a new source file it can tell the Geci framework which source set to create the new file. For example the sample XML bean generator reads an XML file which is in the resources but it wants to generate Java code in the main Java sources directory.

This error happens when a generator wants to generate a new source file in a named set but that set is not defined in the test fluent generator expression. The name of the source set the generator uses can be hard wired into the generator or the generator may use the configuration (either annotation, comment string or the segment start in the source file).

Global segment was opened when the there were already opened segments
Segment was opened after the global segment was already created.

These errors are certainly an error in one of the generators executing in the test phase. Generators can write certain parts of a source file. These are the parts that are between

//<editor-fold ...>

and

//</editor-fold>

lines. Generators can also write the whole source code. This is usually done when the generator creates a new source code that is totally generated without any manually edited part in it. In such a case the generator is opening the so called "global segment" of the source code. A generator should never open a named segment and the global segment the same time. Even different generators should not do that. That way one change would simply overwrite the other change. When this happens the Geci framework throws one of these exceptions.

Segment "…" disappeared from source "file name"

This error is certainly an error in one of the generators executing in the test phase. When this happens it means that the code generation could find a named segment when one of the generator wanted to write code into it, but the source code does not have this segment any more when the changes are to be written back. This may happen if one of the generators changes the source object directly. It may also be an internal error in the framework.

Source directory [ list of directories ] is not found

This error happens when the framework cannot find a configured source set. The source sets are configured with alternative directories. This is to help he framework to be executed from different working directories (CWD, current working directory), which usually happens in case of multi module projects. The error message lists all the directories where it tried to find the source set, but it could not.

If this error happens you should check the configuration in the method call source() in the test and you should also check which CWD the code generation execution was started in.

None of the configured directories { [ … ] } are found

This is a very similar error as the previous one. This error is thrown if none of the configured source sets can be found. When you configure source sets you can use sensible default. For example leaving the call to source() out of your configuration chain will configure all the four maven source sets (test vs. main and java vs. resource) for you automatically. You can use this configuration when you do not have one or more of the four sets in your project. When some of the sensible defaults are used then the source set discovery works in a lenient mode. It is not an error if some of the source sets are not defined. In that the previous error will not be displayed. However, it is a serious issue if none of the source sets can be found. In that case there is no source to work on. Also, when you define a source set yourself and not using defaults then it has to be there on the disk.

If you see this error check the CWD if that is the project root or the root of some module project.

FilesExpand file tree

REFERENCE.adoc

Latest commit

History