Home

FlatMapper

FlatMapper is a library to import and export data from and to plain text files.

Plain text files are still very widely used on legacy systems and a favorite format for Human-to-System interface. This project was born from a need to read and write data from plain text files. I needed a lightweight module that would do just that, no extra needless weight. Since most of the libraries that I found at the time were either code intrusive, or had extra dependencies, I decided to write my own.

My goal was to write it with a nice fluent API, must be simple to use, minimal dependencies and dependents and fast! I had a great time developing it (I still do).

Since the project that originated this library, I already used it again on 2 other projects.

Key Features

• Fast - Only uses Static Reflection and Dynamic methods
• It supports character delimited and fixed length files
• Non intrusive - You don't have to change your code. Any POCO will work
• No external Dependencies
• Iterative reads - Doesn't need to load the entire file into memory
• Multi-line support - Only if character delimited and quoted
• Nullables support
• Fluent Interface
• Per line Error handling
• Simple to use

How to use

Before we start reading or writing from files, we need to specify the layout of the file. We only need to do this once. Imagine the following scenario. We need to read and write from text files to and from this class:

public class TestObject
{
    public int Id { get; set; }
    public string Description { get; set; }
    public int? NullableInt { get; set; }
}

In the following sections, you'll find out how to setup the file layout with both fixed length and character delimited.

Fixed Length Layout

var layout = new Layout<TestObject>.FixedLengthLayout()
				.HeaderLines(1)
				.WithMember(o => o.Id, set => set.WithLenght(5).WithLeftPadding('0'))
				.WithMember(o => o.Description, set => set.WithLenght(25).WithRightPadding(' '))
				.WithMember(o => o.NullableInt, set => set.WithLenght(5).AllowNull("=Null").WithLeftPadding('0'));

Delimited Layout

var layout = new Layout<TestObject>.DelimitedLayout()
	            .WithDelimiter(";")
	            .WithQuote("\"")
				.HeaderLines(1)
	            .WithMember(o => o.Id, set => set.WithLenght(5).WithLeftPadding('0'))
	            .WithMember(o => o.Description, set => set.WithLenght(25).WithRightPadding(' '))
	            .WithMember(o => o.NullableInt, set => set.WithLenght(5).AllowNull("=Null").WithLeftPadding('0'));

With this setup is also possible to have multi-line fields, as long they are Quoted.

Reading and Writing

The reading is interactive, meaning that only when a new item is requested, the data will be read. This helps avoiding reading the entire file into memory and only then parsing the data. Data is parsed on demand.

This library connects into the Stream class of the core framework. This way, don't have restrictions in the encoding and it's outside the scope of the library to free any resource.

//Reading data
using (var fileStream = File.OpenRead("c:\temp\data.txt"))
{
    var flatfile = new FlatFile<TestObject>(layout, fileStream);
    foreach(var objectInstance in flatfile.Read())
    {
        //Do Somethig....
    }
}

//Writing data
using (var fileStream = File.OpenWrite("c:\temp\data.txt"))
{
    var flatfile = new FlatFile<TestObject>(layout, fileStream);
    flatfile.Write(listOfObjects);
}

Error Handling

Optionally, per line/object instance you can control the behavior if any error is thrown due to some unexpected format or any other error, for that matter. By specifying a Func<string, Exception, bool> into the handleEntryReadError parameter of the constructor of FlatFile<T>, every-time any input error occurs, that function is executed, with the line and the Exception that was thrown. If that function returns true the Exception is ignored, and the import continues. If not, the Exception that originated the call, will be re-thrown.

private bool HandleEntryReadError(string line, Exception exception)
{
    Log.LogError("Error reading line :" + line, exception);
    return true;
}

/* ... */
var flatfile = new FlatFile<TestObject>(layout, fileStream, HandleEntryReadError);

Performance

No performance tests were done yet.

The Road to 1.0

Before the 1.0 release, there is some stuff that I would like to present in the library.

• Async (for this to actually work, it will need to work by blocks, dropping the Iterative reads feature)
• Better support for more core types (especially DateTime)
• Support for custom converters