updating to 0.2.0, applying bugixes, doc updates, unit coverage

Author: steveash

.editorconfig

@@ -0,0 +1,7 @@
+[*]
+charset=utf-8
+end_of_line=lf
+insert_final_newline=true
+indent_style=space
+indent_size=2
+

.gitignore

@@ -14,3 +14,5 @@
 # virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
 hs_err_pid*
 target
+build
+classes

README.md

@@ -1,37 +1,77 @@
-jopenfst
+
+JOpenFST
 ========
 
-A partial Java port of the C++ [OpenFST library](http://www.openfst.org/twiki/bin/view/FST/WebHome) which provides a library to
- build weighted finite state transducers and perform various common FST tasks such as:
+A _partial_ Java port of the C++ [OpenFST library](http://www.openfst.org/twiki/bin/view/FST/WebHome), which provides 
+a library to build weighted finite state transducers (WFSTs) and perform various common FST/FSA tasks such as:
 
-* Determinization
+* Determinization (for both acceptors and transducers)
 * Union / Intersection 
 * Composition
 * Shortest Path computation
- 
-This project was originally forked from the CMU Sphinx project.  This was originally work by John 
-Salatas as part of his GSOC 2012 project to port phonetisaurus over to java.  Since then the code appears to be 
-abandoned and doesn't appear to have been integrated in to the final CMU Sphinx project trunk.  I needed a decent 
-WFST library for some of my stuff, so I used his code as a starting point. I have cleaned up quite a bit
- of the code, really changed the APIs, and updated unit tests. My JG2P project uses this, and thus I have
- some confidence that the code is working accurately. However, I would still consider this *beta quality*. When I am comfortable with the stability, I will push a v1.0 to Maven Central Repo.
+
+OpenFST is a mature, elegant, and feature rich library in C++. JOpenFST aims to implement many of the features of 
+OpenFST in a pure Java implementation, which can be useful if you are trying to use WFST operations within an existing
+pure Java architecture or service. Some environments are not easily suited to using JNI or creating a separate C++ 
+service endpoint to do all of the WFST operations. In these circumstances, JOpenFST provides an alternative.
+
+OpenFST is designed quite elegantly and relies on sophisticated C++ template metaprogramming features to achieve 
+top speed (and nice generality). Since Java offers no rich metaprogramming facility, JOpenFST differs significantly
+in its API and implementation. JOpenFST is probably more fairly described as a re-imagining of OpenFST within the 
+constraints and idioms of Java/JVM. Because of that, JOpenFST lacks some features that are present in OpenFST, but 
+hopefully that gap will close over time (PRs welcome!).
+
+Here are some of the most notable differences from OpenFST:
+* All WFST operations are eagerly executed, there is no deferred/lazy evaluation. There are however some optimizations
+  when doing operations on Immutable instances (see Compose) to avoid unnecessary copying.
+* JOpenFST can only import/export using the OpenFST/AT&T text format (as produced by `fstprint` and consumed by 
+  `fstcompile`); JOpenFST cannot currently import OpenFST binary models (as produced by `fstcompile`).
+* There are mutable and immutable types that mirror each other (MutableFst, ImmutableFst, MutableState, ImmutableState, etc.)
+* There is a Gallic weight and semiring but not a separate String semiring.
+* The Gallic Weights are either Gallic Restricted or Gallic Min; if you want General Gallic weights, you have to use the
+  Union Semiring directly.
+* The following operations are implemented:
+    * ArcSort
+    * Compose
+    * Connect
+    * Determinize (for both acceptors and transducers; all modes: functional, non-functional, and disambiguate)
+    * Shortest Paths
+    * Project
+    * Remove Epsilon
+    * Reverse
+* The following operations are currently NOT implemented (PRs welcome):
+    * Minimization (coming soon)
+    * TopSort (coming soon)
+    * Closure
+    * Concat/Union 
+    * Encode/Decode
+    * Difference/Intersect
+    * Invert
+    * Prune (as a separate operation)
+    * Push
+    * Synchronize
+    
+This project was originally work in the CMU Sphinx project by John Salatas as part of his GSOC 2012 project, but since
+then it has been mostly rewritten to bring in new enhancements, improve the API, and improve performance. 
 
 Current version:
 ```xml
 <dependency>
     <groupId>com.github.steveash.jopenfst</groupId>
     <artifactId>jopenfst</artifactId>
-    <version>0.1.1.ALPHA</version>
+    <version>0.3.0</version>
 </dependency>
 ```
 
 Quick Start
 -----------
-The API started out pretty close to OpenFST but is diverging over time. The basic abstractions of `Fst`, `State`, `Arc`,
-and `SymbolTable` have conceptual analogs in OpenFST. In jopenfst there are *Mutable* and *Immutable* implementations
+The API started out pretty close to OpenFST but has diverged over time. The basic abstractions of `Fst`, `State`, `Arc`,
+and `SymbolTable` have conceptual analogs in OpenFST. In JOpenFST there are *Mutable* and *Immutable* implementations
 of each. As you programmatically build up your WFSTs, you will use the Mutable API.  If you want to de/serialize larger
-models (large WFSTs built from training data that are used to construct lattices) and these models don't need to change, then you can convert the mutable instance into an immutable instance after you are done building it (`new ImmutableFst(myMutableFst)`.
-ImmutableFsts are likely faster at some operations and also are smarter about reducing unnecessary copying of state.
+models (large WFSTs built from training data that are used to construct lattices) and these models don't need to 
+change, then you can convert the mutable instance into an immutable instance after you are done building it 
+(`new ImmutableFst(myMutableFst)`. ImmutableFsts are likely faster at some operations and also are smarter about 
+reducing unnecessary copying of state.
 
 The MutableFst API is probably the bast place to start. Here is a sample showing how to
 construct a WFST which shows the basic operations of fsts, states, arcs, and symbols.
@@ -61,33 +101,34 @@ fst.addArc(startState, "inC", "outD", fst.getOrNewState("state3"), 123.0);
 ```
 Input and Output
 ----------------
-jOpenFst supports reading/writing the OpenFst text format and our own jopenfst binary serialization format (more compact than text). We cannot currently read/write openfsts binary serialization format, though pull requests with that functionality are very welcome.
+jOpenFst supports reading/writing the OpenFst text format and our own JOpenFST binary serialization format 
+(more compact than text). We cannot currently read/write OpenFSTs binary serialization format.
 
-To read OpenFst text format, you need a `mymodel.fst.txt` file that describes all of the arcs and weights. If you are using labeled states, inputs, or outputs (e.g. for a transducer) then you also need files for those named `mymodel.input.syms`, `mymodel.output.syms`, and `mymodel.states.syms` respectively. An exmaple of these files is in the `src/test/resources/data/openfst` folder in the source.
+To read OpenFst text format, you need a `mymodel.fst.txt` file that describes all of the arcs and weights. If you 
+are using labeled states, inputs, or outputs (e.g. for a transducer) then you also need files for those 
+named `mymodel.input.syms`, `mymodel.output.syms`, and `mymodel.states.syms` respectively. An exmaple of these files 
+is in the `src/test/resources/data/openfst` folder in the source.
 
-To read/write the text format call methods `Convert.importFst(..)` and `Convert.export(..)`. Both of these return instances of `MutableFst` which can be converted into `ImmutableFst` via `new ImmutableFst(myMutableFst)`.   There are importFst overloads for dealing with either Files or resources from the classpath.
+To read/write the text format call methods `Convert.importFst(..)` and `Convert.export(..)`. Both of these return 
+instances of `MutableFst` which can be converted into `ImmutableFst` via `new ImmutableFst(myMutableFst)`.   
+There are importFst overloads for dealing with either Files or resources from the classpath.
 
-To read/write the binary format call methods `FstInputOutput.readFstFromBinaryFile` and `FstInputOutput.writeFstToBinaryFile` (there are overloads for dealing with streams/resources.  Resources are useful if you want to package your serialized model in your jar and just read it from the classpath.
+To read/write the binary format call methods `FstInputOutput.readFstFromBinaryFile` and 
+`FstInputOutput.writeFstToBinaryFile` (there are overloads for dealing with streams/resources.  
+Resources are useful if you want to package your serialized model in your jar and just read it from the classpath.
 
 Resources
 ---------
 
 * [John Salatas' blog](http://jsalatas.ictpro.gr/tag/java-fst/) has some posts that describe some of his initial design 
-decisions.  I imagine that as I work on this these blog posts will become less representative of jopenFST but for the 
-moment its pretty close
+decisions.  The library has diverged pretty significantly from this original version, but this is still a reference.
 * [C++ OpenFST library](http://www.openfst.org/twiki/bin/view/FST/WebHome) describes some of the FST algorithms implemented.
 
-Changes:
+Release History
 ------------
-
-* Adding back edges (kind of) to dramatically optimize a number of the original implementations that had poor algorithmic complexity
-** In my jg2p project this reduced runtime for my datasets by a factor of 30x
-* The original Connect/Trim implementation was wrong; fixed now.
-* Separated out interfaces for read-only/writeable elements (Arcs, States, Fsts) which allows
-convenient things like "union" symbol tables (to do mutating things without copying the entire source symbl table)
-* Clearly separated out algorithms by ones that mutate input args vs ones that produce new fsts
-* Updated some IO routines, used exceptions instead of System.err logging, some cleanup, fixed unit tests
-* Changed packages (although it can still deserialize FST models from the original repo)
+* 0.3.0 - Adding union semiring, gallic semiring, and implemented Determinization for transducers with multiple modes
+(functional, nonfunctional, and disambiguate) to match the behavior and options available in OpenFST.
+* 0.2.0 - code covering unit tests, improvements to interoperability of the text format between JOpenFST and OpenFST.
 
 
 

pom.xml

@@ -22,8 +22,8 @@
   <groupId>com.github.steveash.jopenfst</groupId>
   <artifactId>jopenfst</artifactId>
   <name>jopenfst</name>
-  <version>0.1.1.ALPHA</version>
-  <description>Java port of the OpenFST library; forked from the CMU Sphinx project</description>
+  <version>0.2.0</version>
+  <description>Partial Java port of the OpenFST library; forked from the CMU Sphinx project</description>
   <packaging>jar</packaging>
 
   <url>https://github.com/steveash/jopenfst</url>
@@ -124,8 +124,8 @@
         <artifactId>maven-compiler-plugin</artifactId>
         <version>3.1</version>
         <configuration>
-          <source>1.7</source>
-          <target>1.7</target>
+          <source>1.8</source>
+          <target>1.8</target>
         </configuration>
       </plugin>
       <plugin>
@@ -163,6 +163,9 @@
             <goals>
               <goal>jar</goal>
             </goals>
+	    <configuration>
+	      <additionalparam>${javadoc.opts}</additionalparam>
+	    </configuration>
           </execution>
         </executions>
       </plugin>
@@ -173,6 +176,15 @@
     </plugins>
   </build>
   <profiles>
+    <profile>
+      <id>java8-doclint-disabled</id>
+      <activation>
+        <jdk>[1.8,)</jdk>
+      </activation>
+      <properties>
+        <javadoc.opts>-Xdoclint:none</javadoc.opts>
+      </properties>
+    </profile>
 
     <profile>
       <id>javadoc</id>

src/main/java/com/github/steveash/jopenfst/AbstractSymbolTable.java

@@ -16,19 +16,23 @@
 
 package com.github.steveash.jopenfst;
 
-import com.google.common.base.Function;
-import com.google.common.collect.Iterables;
-
 import com.carrotsearch.hppc.IntObjectOpenHashMap;
 import com.carrotsearch.hppc.ObjectIntOpenHashMap;
 import com.carrotsearch.hppc.cursors.IntCursor;
 import com.carrotsearch.hppc.cursors.ObjectCursor;
 import com.carrotsearch.hppc.cursors.ObjectIntCursor;
 import com.github.steveash.jopenfst.utils.FstUtils;
+import com.google.common.base.Function;
+import com.google.common.collect.Iterables;
 
 import java.util.Iterator;
 
 /**
+ * The base abstract implementation which uses carrotsearch primitive maps for optimized mappings of
+ * int -> string and vice versa.
+ * This implementation is effectively thread safe if no mutating operations are performed after
+ * construction.
+ *
  * @author Steve Ash
  */
 public abstract class AbstractSymbolTable implements SymbolTable {
@@ -41,6 +45,11 @@ public String apply(ObjectCursor<String> input) {
         }
       };
 
+  /**
+   * Returns the current max id mapped in this symbol table or 0 if this has no mappings
+   * @param table
+   * @return
+   */
   public static int maxIdIn(SymbolTable table) {
     int max = 0;
     for (ObjectIntCursor<String> cursor : table) {

src/main/java/com/github/steveash/jopenfst/Arc.java

@@ -17,15 +17,37 @@
 package com.github.steveash.jopenfst;
 
 /**
+ * Interface for the contract of an Arc in an FST
+ * @see ImmutableArc
+ * @see MutableArc
  * @author Steve Ash
  */
 public interface Arc {
 
+  /**
+   * Get the weight of this edge in the FST (range of values depends on the Semiring used for the FST)
+   * @see Fst#getSemiring()
+   * @return
+   */
   double getWeight();
 
+  /**
+   * Get the index of the input symbol for this edge of the fst
+   * @return
+   */
   int getIlabel();
 
+  /**
+   * Get the index of the output symbol for this edge of the fst
+   * @return
+   */
   int getOlabel();
 
+  /**
+   * Get the reference to the next state in the FST; note that you get call `getNextState().getId()` to get the
+   * FST state id for that state but some operations will be constructing new results and state ids will not be
+   * consistent across them (obviously). If you are using state symbols/labels then the labels will be constistent
+   * @return
+   */
   State getNextState();
 }