Peon

A data pipeline tool written in Rust.

• Given a collection of loose and unorganized files will match and group them into a structured destination folder.
• Will build files according to matched rule.
• Only build files that have changed.
• Support includes A->B->C.
• Clean destination folder of orphaned files.

Configuration

By convention .peon-config.json will be loaded from working directory. This can be overloaded with --config.

{
    "ignores": [ <Glob Pattern> ],
    "rules": [ <Rule> ]
}

Rule

{
    "ruleName": "string",
    "matcher": <Matcher>,
    "builder": <Builder"
}

File Extension Matcher

{
    "type": "fileExtension",
    "fileExtensions": [ <string> ]    
}

Examples

All examples should be run from their folder to pickup the appropriate peon-config.json by convention. Will also write peon.history to that folder by convention.

Builders

CopyFile

The simplest builder, will copy file and nothing else.

Rule

{
    "ruleName": "CopyTxt",
    "matcher": {
        "type": "fileExtension",
        "fileExtensions": [
            "txt"
        ]    
    },
    "builder": {
        "type": "copyFile",
        "destinationFolder": "txt"
    }
}  

Example

<root>\peon\examples\copy_files> cargo run -- --source src --destination dst

CopyJsonFile

Copy file with optional JsonSchema validation. If schemaFileName is configured then --json-schema-folder-path must be provided when running.

Rule

{
    "ruleName": "CopyPlayer",
    "matcher": {
        "type": "fileExtension",
        "fileExtensions": [
            "player"
        ]    
    },
    "builder": {
        "type": "copyJsonFile",
        "destinationFolder": "entities",
        "schemaFileName": "player-schema.json"                
    }
}

Example

<root>\peon\examples\copy_json_files> cargo run -- --source src --destination dst --json-schema-folder-path schemas

CompileShader

Compile shaders using Fxc. Will handle #include.

Rule

{
    "ruleName": "CompileShader",
    "matcher": {
        "type": "fileExtension",
        "fileExtensions": [
            "hlsl"
        ]
    },
    "builder": {
        "type": "compileShader",
        "destinationFolder": "shaders",
        "destinationExtension": "fxco",
        "fxcExeName": "fxc.exe",
        "vertexShader": {
            "postfixes": ["_vs", "VS"],
            "profile": "vs_5_0"
        },
        "pixelShader": {
            "postfixes": ["_ps", "PS"],
            "profile": "ps_5_0"
        },
        "computeShader": {
            "postfixes": ["_cs"],
            "profile": "cs_5_0"
        },
        "enableStrictMode": true,
        "enableWarningsAsErrors": true,
        "enableDebuggingInformation": false,
        "enableOptimizations": true
    }
}

Example

Download and place fxc.exe from Fxc into compile_shaders/tools folder.

<root>\peon\examples\compile_shaders> cargo run -- --source src --destination dst --tool-folder-path tools

ConvertTexture

Resolve texture type and run Texconv or Cmft with appropriate flags.

Texture types are resolved through a number of rules:

1. folder level attributes stored in .texconv file. This file contains glob patterns for the folder to resolve the texture type.

{
    "*.png": "Particle",
    "*.tga": "Particle"
}

2. side-by-side .mesh_material files contain explicit texture names.

{
    "base_color_texture": "foo_clr",
    "occlusion_roughness_metallic_texture": "foo_orm",
    "normal_texture": "foo_nrm",
	"mask_texture": "foo_msk"
}

Rule

{
    "ruleName": "ConvertTexture",
    "matcher": {
        "type": "fileExtension",
        "fileExtensions": [
            "png",
            "tga",
            "smaa_dds"
        ]
    },
    "builder": {
        "type": "convertTexture",
        "destinationFolder": "textures",
        "destinationExtension": "dds",
        "texConv": {
            "exeName": "texconv.exe",
            "formats": {
                "StandardColor": "BC7_UNORM",
                "MeshColor": "BC7_UNORM",
                "MeshOrm": "BC7_UNORM",
                "MeshNormal": "BC5_SNORM",
                "MeshMask": "BC7_UNORM",
                "Particle": "BC7_UNORM",
                "SMAAArea": "BC5_UNORM",
                "SMAASearch": "BC4_UNORM"
            }
        },
        "cmft": {
            "exeName": "cmft.exe",
            "textureTypes": [
                "SkyboxPanoramicColor"
            ]
        }
    }
}

Example

Download and place texconv.exe from Texconv into convert_textures/tools folder.

<root>\peon\examples\convert_textures> cargo run -- --source src --destination dst --tool-folder-path tools

RunTool

Run arbitrary tool exe that is expected to generate destination file. toolArguments can contain templates that will be substituted at runtime : [source_file, destination_file, destination_folder]

Rule

{
    "ruleName": "RunTool",
    "matcher": {
        "type": "fileExtension",
        "fileExtensions": [
            "png"
        ]
    },
    "builder": {
        "type": "runTool",
        "destinationFolder": "tool_output",
        "destinationExtension": "dds",
        "toolName": "texconv.exe",
        "toolArguments": ["-f", "BC7_UNORM", "-y", "{source_file}", "-o", "{destination_folder}"]
    }
}

Example

Download and place texconv.exe from Texconv into run_tool/tools folder.

<root>\peon\examples\run_tool> cargo run -- --source src --destination dst --tool-folder-path tools

ZipFolder

Zip a collection of source files into a single destination file.

Rule

{
    "ruleName": "ZipFolder",
    "matcher": {
        "type": "folderName",
        "folderName": "foo"
    },
    "builder": {
        "type": "zipFolder",
        "destinationFolder": "zipped_folders",
        "destinationExtension": ".zip"
    }
}

Example

<root>\peon\examples\zip_folder> cargo run -- --source src --destination dst

Running Tests

cargo test

or

cargo nextest run