v3.7.1

What's Changed

Fixes

• --template-append (#133, fixes #132) — opt-in CLI flag (also output.template_append: true in the YAML config) that switches all configured --templateOutput files to append mode for that run. Restores the pre-3.0 behavior for users who build cumulative NDJSON feeds for Splunk/ELK across multiple runs. The default remains overwrite, so single-document exports such as the ATT&CK Navigator layer or SARIF stay valid.

Internal

• Hardened Dockerfile: dropped the unused git apt step and the image now runs as a non-root user.

Usage

python zircolite.py --evtx logs/ --ruleset rules/rules_windows_generic.json \
    --template templates/exportForSplunk.tmpl \
    --templateOutput cumulative.ndjson \
    --template-append

output:
  templates:
    - template: templates/exportForSplunk.tmpl
      output: cumulative.ndjson
  template_append: true

Warning

Append mode is intended for line-oriented templates (exportForSplunk.tmpl, exportForELK.tmpl, exportForTimesketch.tmpl, exportNDJSON.tmpl). It is not appropriate for templates that produce a single JSON document, such as exportForAttackNavigator.tmpl or exportForSARIF.tmpl.

Full Changelog: v3.7.0...v3.7.1

v3.7.0

What's Changed

New Features

• Graceful Ctrl+C shutdown — two-stage interrupt handling: first Ctrl+C finishes in-flight work cleanly, second forces exit
• --auto-index [N] (default 5) — analyzes the loaded ruleset and auto-creates SQLite indices on the top-N most-referenced columns
• ATT&CK Navigator export updated to ATT&CK v18 / Navigator 5.3.1, with tactic information and severity-based color legend
• New zircolite/attack.py module centralizing ATT&CK tag extraction (techniques + tactics, alias-normalized)

Fixes & Improvements

• Shutdown-request checks in parallel and streaming loops for safer cancellation
• More robust database connection handling: clearer type signatures, thread-safe pragma setup, better error handling during connection backup
• Fixed templateOutput initialization when not supplied

Internal

• Consolidated ATT&CK parsing logic
• Type annotations cleanup across core modules

Rulesets

• Refreshed all 12 rules_*.json rulesets

Documentation

• Simplified Advanced.md examples and jq query patterns; condensed transform examples

Full Changelog: v3.6.3...v3.7.0

v3.6.3

What's Changed

• patch streaming.py by @jkaspereit in #130

New Contributors

• @jkaspereit made their first contribution in #130

v3.6.1

What's Changed

• Fix #128 to allow processing of partially corrupted, malformed, or incomplete event log data instead of failing on chunk header parsing errors
• Add --strict flag for EVTX parsing to enforce strict error handling by @wagga40 in #129

Full Changelog: v3.6.0...v3.6.1

v3.6.0

v3.6.0

What's new

Sigma correlation rules

• Time column matches your logs sooner. Before Sigma rules are converted, Zircolite figures out which timestamp field your data uses (when you use defaults and auto-detection). Correlation rules then use that same field, including for Linux and other non-Windows logs—so you get fewer wrong or empty correlation results.
• Correlation-only YAML documents in multi-document Sigma files are recognized when they’re valid correlation rules (not only classic logsource + detection rules).
• Channel/Event ID pre-filtering still speeds up normal rules, but correlation rules no longer disable that optimization for the whole ruleset.
• Rules that exist only to support a correlation rule are still processed so the correlation SQL can be built correctly.

CSV output

• CSV uses one fixed set of columns based on the first detection written. Fields that show up only in later rules are not added as new columns—use JSON if you need the full field set from every rule.
• --csv explains this in the help text.

Upgrade notes

• Need all columns from every rule? Use JSON (or process JSON), not CSV.
• If you turn off auto-detection, set --timefield to the flattened timestamp column name yourself so correlation rules stay consistent with your events.

Full changelog: v3.5.0...v3.6.0

v3.5.0

What's New

Log Ingestion

• Compressed/archived log support — Process logs directly from ZIP and 7z archives (single-file archives supported). Use --archive-password to handle encrypted archives.

Export & Templates

• New built-in templates:
  • exportForAttackNavigator.tmpl → Generate ATT&CK Navigator layers
  • exportForSARIF.tmpl → Export findings in SARIF format
• New shortcut flag --timesketch for faster Timesketch exports
• Bugfix: Resolved an issue affecting Timesketch exports

Rule Development

• Rule tester — Validate rules against a sample file with --test-rules <FILE>
• Rule profiler — Identify slow or expensive rules with --profile-rules

Performance

• Windows Event Logs: Channel field is now indexed, improving filter performance significantly
• SQLite index management: Create or drop indexes on demand via --add-index / --remove-index
• Memory & worker optimizations across the board for faster, leaner processing

UI/UX

• Per-file progress bars — Track processing progress at the individual file level

Rulesets

• Rulesets have been updated

Full Changelog: v3.3.0...v3.5.0

v3.3.0

What's new

New features

• --timesketch shortcut — Use the Timesketch export template and write to timesketch-<RAND>.json in one flag. Multiple exports no longer overwrite each other.
• ECS and Winlogbeat field mappings — New mappings in config/config.yaml for Elastic Common Schema and Winlogbeat (channel, event ID, timestamps, etc.). Thanks to @maspital.
• Incremental result writing — Parallel processing can write detection results incrementally
• Refined parallel worker calculation — Improved choice of worker count

Improvements

• --keepflat — Behavior is now clearly defined: the flattened JSONL contains only events that were processed (events dropped by early event filtering or --after/--before are excluded). Combine with --no-event-filter to include all events.
• Early event filtering — Documentation (Advanced.md, Usage.md) updated to explain when filtering is enabled, how Channel/EventID logic works, and how it interacts with keepflat and time range.
• Taskfile — Cleanup task improvements.
• Rulesets — Updated Linux and Windows rulesets and added rules for suspicious activities.

Full Changelog: v3.2.0...v3.3.0

v3.2.0

What's Changed

• UI/UX : Lot of enhancements in the UI/UX
• Transforms: New transforms added. Now transforms can be enabled with --all-transforms, --transform-category, --transform-list
• CSV output: Correct bug #114
• Config:config/config.yaml is the default/canonical config; config/fieldMappings.yaml is deprecated (still supported, may be removed later). Docs and examples now reference config.yaml
• Rulesets: Windows and Linux rulesets updated

Full Changelog: v3.1.0...v3.2.0

v3.1.0

What's Changed

New

• Log type detector (zircolite/detector.py): content-based detection of log format and timestamp field so users need fewer explicit flags (--json-input, --auditd-input, etc.). Supports EVTX (binary/JSON/XML), Sysmon for Linux, Auditd, CSV, ECS/Elastic JSON, EVTXtract output, and generic JSON/JSONL.
• New UI/UX: the UI/UX has been updated to make a better use of the Rich library.

Updated

• Rules: Windows generic and Sysmon rulesets (including high/medium variants).
• Config: config/fieldMappings.yaml and related config handling.
• Docs: Advanced, Usage, Internals, README, and sidebar.

Full Changelog: v3.0.2...v3.1.0

v3.0.2

What's New in v3.0.2

• Correct handling of order and priority of pipelines #110 Thanks to @Maspital

v3.0.0 changes :

TL; DR;

• Huge speed improvements
• New transforms
• New UI
• Based on new pySigma backend

Long version

• Package refactor: Core logic split into zircolite package (core, streaming, flattener, rules, config_loader, console, parallel, etc.).
• Event filter: Early skip by channel/EventID from rules before flattening.
• Field transforms: Python transforms and alias fields in fieldMappings.yaml.
• YAML-only field mappings: fieldMappings.json removed.
• Rules v2: Zircolite-Rules-v2; dropped _full and _pysigma rulesets.
• YAML run config: Full config via YAML (see config/zircolite_example.yaml).
• Taskfile: Docker, rules update, cleanup tasks.
• Rich console: Improved progress and logging.
• Bug fix: Multiple rule files no longer produce invalid JSON (missing commas).
• Docs and tests: Updated docs; new tests for event filter and transforms.