CHANGELOG.md

Changelog

0.13.1 (2026-07-31)

Features

• app: add the plan Ordering as an http header to the response (4b501d5)
• query_engine: add a specialized OrderByWithLimit node (34d3814)
• query_engine: implement free IUs for boolean expressions (#1377) (f366c81)

Bug Fixes

• silo: allow limit on groupBy (035e244)

0.13.0 (2026-07-29)

⚠ BREAKING CHANGES

• silo: Insertions response: removed insertion field. Mutations response: removed mutation field. The string was constructed from information that is still present in the response.
• preprocessing: the database config options defaultNucleotideSequence and defaultAminoAcidSequence have been removed. sequenceName is now required on all nucleotide and amino acid filters (nucleotideEquals/aminoAcidEquals, hasMutation/hasAAMutation, insertionContains/aminoAcidInsertionContains, symbolInSet and the mutation profile filters).

Features

• app: add a --version command to the silo binary (8b3146e)
• preprocessing: remove default sequence names from database config (#1386) (b8712ec)
• query_engine: add join operator (f7cbaf4)
• silo: allow updating of string columns (818a737)
• silo: remove insertion and mutation field from responses (#1408) (72fca9f)

0.12.1 (2026-07-27)

Features

• performance: add co-occurrence query benchmark (2c52241)
• query_engine: bitmap-based aggregation for count group-bys (a81fd24)
• SaneQL: pipeline optimization step: map pull up through filter (#1354) (b1966c4)
• silo: add update function to python bindings (6e27cd5)
• silo: add WebAssembly build (#1320) (f41993a)

Bug Fixes

• silo: sort nulls consistently as smallest value (00420f3)

Performance Improvements

• silo: always prime Selection with most-selective predicate (a8b458e)
• silo: when adapting references, compute the coverage bitmaps only when necessary (8f178dc)

0.12.0 (2026-07-15)

⚠ BREAKING CHANGES

• preprocessing: remove deprecated preprocessing config values intermediateResultsDirectory, preprocessingDatabaseLocation, duckdbMemoryLimitInG and database config values dateToSortBy, partitionBy
• silo: use date32 type for dates instead of converting to string
• silo: change query interface to saneql

Features

• add unionAll operator to SaneQL queries (#1305) (53da65f)
• preprocessing: remove deprecated preprocessing config and database config values (#1287) (fbe4014)
• SaneQL: add schema operator to get query result schema (#1333) (f61955e)
• silo: add a dedicated filter-pushdown-pass and move node resolution to own file (2da2f48)
• silo: add a references to input fields in .map(...) expressions (2ce79f2)
• silo: add an at expression (82f98b4)
• silo: add column narrowing to query planner (90071f3)
• silo: change query interface to saneql (daba3ea)
• silo: log the query plan and optimization steps (50e3e17)
• silo: organize string columns in immutable chunks (b8922e4)
• silo: use date32 type for dates instead of converting to string (f907787)

Bug Fixes

• preprocessing: ignore bracketed comments in Newick tree files (#1268) (9d0e323)
• silo: disambiguate unary/binary minus in SaneQL (#1266) (e1739ff)
• silo: null sequences should not match N symbol queries (004bba4)

Performance Improvements

• silo: replace charToSymbol switch with a lookup table (4395af5)

0.11.3 (2026-05-18)

Features

• silo: add a MutationProfile filter (2d49d9b)

0.11.2 (2026-04-17)

Features

• silo: add option to ignore unknown lineage values (16e1389)

0.11.1 (2026-03-23)

Features

• silo: allow sequence-column inputs to be zstd-compressed in base64 format (1f2bbc5)

Bug Fixes

• silo: let N (X for amino acids) also code for - (a081630)

0.11.0 (2026-03-03)

⚠ BREAKING CHANGES

• silo: "" is no longer considered null for date columns

Features

• silo: add a DateEquals expression (9cf5fd4)

Bug Fixes

• silo: actually validate dates on insertion (9e24c7b)

0.10.1 (2026-02-27)

Features

• performance: add performance benchmark for the many short-read filters (f8ef7ab)
• silo: add verbosity argument to CLI interface (0bc682d)

0.10.0 (2026-02-22)

⚠ BREAKING CHANGES

• silo: SILO will return null instead of NN...NN/XX...XX for sequences that were null in the input. "" in the input is no longer considered null for columns with type: string and generateIndex: true

Features

• benchmarking: remove keep-silo feature (e1486ae)
• silo: add null_bitmap for all columns to explicitly model NULL (baef7d3)
• silo: improve IsNull expression to support IndexedStringColumn, ZstdCompressedStringColumn and SequenceColumn (6e48abf)

0.9.12 (2026-02-19)

Features

• silo: return query result as arrow IPC if the http header is set (6044c73)

0.9.11 (2026-02-18)

Bug Fixes

• silo: error if a user tries to insert invalid insertion positions (ad7ef81)

0.9.10 (2026-02-12)

Features

• silodb: release python bindings as release artifacts (1117dde)

0.9.9 (2026-02-11)

Features

• silo: add a arrow query interface to the python bindings (4b7df7e)
• silo: add python bindings (bb7ca5a)

0.9.8 (2026-02-04)

Bug Fixes

• silo: avoid race-condition in plan clean-up leading to crashes (0685ad8)

0.9.7 (2026-01-30)

Features

• add compiled binaries for linux to GitHub releases (#1129) (c7dd359), closes #575
• ci: add a ci formatting job for workflow/*.yml (a67ed2c)

0.9.6 (2026-01-28)

Features

• performance-tests: add benchmark for nested ors (34d6986)
• silo: add a StringInSet expression to filter for many strings (b67fd5a)
• silo: add an IsNull filter (3437b3d)
• silo: improve rewriting of nested Or expressions (24b27a4)
• silo: rewrite StringEquals and StringInSet expressions during optimization (23553ae)
• test: add performance benchmark for many StringEquals filters (579e6c6)

0.9.5 (2026-01-22)

Features

• build: change builder base image to debian (9e07f48)

Bug Fixes

• silo: correctly adapt local reference in incremental appends (fa2ef25)

0.9.4 (2026-01-15)

Bug Fixes

• legacyNdjsonTransformer: correctly parse floats with serde_json by setting the feature flag float_roundtrip (8d38b27)
• silo: do not use mimalloc on Mac (6d78de8)
• silo: only link mimalloc into the final executable, not the library, which might be included differently (51fa4dc)

0.9.3 (2025-12-16)

Features

• benchmarking: add optional QUERIES custom variable (772cd36)
• benchmarking: allow avoidance of preprocessing by setting OUTPUT_DIR (74c8b84)
• benchmarking: optionally store the produced binaries (7bb482c), closes #1087
• benchmarking: update api-query to ignore empty ignore regex files (f572701)
• silo: add function to dynamically create tables in a silo instance (b70f1df)
• silo: use mimalloc as the default allocator (2203db9)

Bug Fixes

• benchmarking: accept differences in HTTP error responses (df3e18b)
• benchmarking: set a timeout waiting for SILO to start (f50e38d)
• benchmarking: set default for QUERIES env var, not just QUERIES_DIR (20ad729)

0.9.2 (2025-12-03)

Features

• benchmarking: add DEBUG custom variable for running debug builds (541afc8)
• benchmarking: update api-query, enable CSV log file w/ checksum comparisons (3d135d0)
• build: add Makefile rule to generate conanprofile (1a8a334)
• build: tag images with branch name again (670817c)
• ci: add CI workflow that also runs the test suite with debug build (9bdc52b)
• documentation: add documentation for sequence storage format (47e4081)
• silo: add efficient way of computing vertical coverage bitmaps and use them to adapt reference positions (8e7e9f4)
• silo: add reordering of predicates to apply most selective predicates first (1db8cf0)
• silo: better compression for sequences (9eb69c7)
• silo: change IsInCoveredRegion to predicate instead of being its own filter-operator (6fef558)
• silo: enable rewrite of SymbolInSet expressions under Or (5489117)
• silo: split compile function into separate rewrite step (27b7e7c)

Bug Fixes

• build: correctly fix the cppstd for installing the arrow dependency (0c4aed3)
• build: fix Makefile to not repeatedly invoke conan install (346e6a9)
• silo: do not add empty coverage bitmaps (5bd32c7)
• silo: remove memory leak in sequence storage (1da9ecd)

0.9.1 (2025-10-28)

Features

• benchmarking: support versioned datasets (#1031) (4a7399b), closes #1018
• performance: add performance benchmark subdirectory with a benchmark on mutation queries (23af5c1)
• silo: add amino acid codes J, O, U and clean some code (ccf15ed)
• silo: add template to run ppx instances (#1019) (129b428)
• silo: address memory leak concerns (#1008) (7359eea)
• silo: fix clang-tidy configs (no comment allowed in checks), error on warnings in CMake clang-tidy option (#1009) (d5bbaba)
• silo: improve error message if primary key is of type lineage (#1030) (a861141)

Bug Fixes

• e2e: remove obsolete version tag in docker compose files leading to warning (5beabad)
• silo: correctly error when requested insertion positions are larger than reference sequence size (befbc76)
• silo: optimize check for unique keys (4738a27)

0.9.0 (2025-10-03)

⚠ BREAKING CHANGES

• silo: Multiple breaking changes to lineage configuration:
  • PreprocessingConfig: lineageDefinitionsFilename renamed to lineageDefinitionFilenames and changed from string to list
  • DatabaseConfig: generateLineageIndex changed from boolean to string (expects lineage definition filename)

Features

• silo: support multiple lineage systems (f4be635)

0.8.6 (2025-09-26)

Features

• improve logging in the query_plan (bd2b494)
• silo: add a health endpoint that checks whether the api is able to answer requests (5422ff1)
• silo: add throttled reslicing of ExecBatches before zstd decompression (8ac2859)
• silo: remove TBB dependency (#985) (91ec897), closes #873
• update arrow to version 21.0.0 (478366e)

Bug Fixes

• improvements of the arrow plan teardown after a query finishes (223e5c6)
• test: properly free request handlers in tests (0f9e0d9)

0.8.5 (2025-09-09)

Features

• silo: add possibility to disable generation of unaligned sequence columns in schema (b3bf856)
• silo: add possibility to specify offset in the input ndjson (bf3a1d9)
• silo: also allow lower-case characters as input in sequences (97bba59)

0.8.4 (2025-09-04)

Features

• bump version manually to release major refactor separately (0a06315)

0.8.3 (2025-09-02)

Features

• default to using number of threads equal to processors (9795d28)

0.8.2 (2025-08-28)

Features

• increase default concurrent connections to 256 and number of worker threads to 8 (987717c)

0.8.1 (2025-08-13)

Features

• add a default timeout of 2 minutes between sending response batches of requests (f5effef)
• add mrcaParent and mrcaDepth to response of MostRecentCommonAncestor queries (f49177b)

0.8.0 (2025-08-11)

⚠ BREAKING CHANGES

• the schema of the expected ndjson file changed. Pass all data flat, and aligned sequence data as a json object together with corresponding insertions. SILO no longer appends "unaligned_" internally when looking for fields in the ndjson

Features

• use simdjson as json parser during preprocessing (542d231)

0.7.11 (2025-08-11)

Features

• silo: include null values in phyloTree column in missingNodeCount (#936) (8f121cf)

0.7.10 (2025-08-07)

Bug Fixes

• bump the silo serialization version (d7adad5)

0.7.9 (2025-08-07)

Features

• benchmarking: add a separate target for the preprocessing (5c18daf), closes #919

Bug Fixes

• benchmarking: check for the presence of the BENCH_OUTPUT_LOG var (d04789b)
• ci: do not cancel in-progress runs, which could lead to non-generation of released images (32a9b83)
• throw an error when a user requests a limit without specifying ordering (3ad8dfe)
• validate a arrow query plan before executing it to catch errors that might lead to dying requests (c49a698)

0.7.8 (2025-08-01)

Features

• silo: store branch lengths and export them in subtree newick (#918) (624f248)

0.7.7 (2025-07-31)

Features

• silo: further optimizations of queries using the phylo tree - use hash sets instead of vectors (#917) (8164013)

0.7.6 (2025-07-29)

Features

• silo: add an endpoint to export a newick of the nodes in the filter (#911) (0bb94b7)
• silo: optimize phyloSubtree action (#916) (7c218d4)
• silo: refactor config - rename phyloTreeNodeIdentifier to isPhyloTreeField (#921) (3085059)

0.7.5 (2025-07-23)

Features

• benchmarking: add benchmarking probes to preprocessing (8fbebe1)
• benchmarking: add probes to "action" part of query evaluation (f63eb41)
• benchmarking: add query type to benchmark context before query execution (e62677c)
• benchmarking: hack: save evobench log file from preprocessing phase (5193ae8)
• benchmarking: optimize & add possibility to pass constexpr string variables (3e7a41b)
• benchmarking: update api-query, add REPEAT, CONCURRENCY vars (4a063d8)
• silo: optimize ndjson sink for higher download speeds (4c97982)

Bug Fixes

• benchmarking: also prefix output dir name with $COMMIT_ID (e18c881)
• benchmarking: prefix output dir name with $DATASET (df6405e)
• build: change format target to also include .hpp files (7c90965)
• project out the superfluous column instead of producing mismatching schema-ExecBatches (53e3187)
• properly designate the hash_count_all output as int64 in the output schema (e04987c)

0.7.4 (2025-07-21)

Bug Fixes

• silo: accept more characters in newick string and improve error messages (#891) (5f8bc9e)
• silo: correctly deduplicate requested fields (78f4146)

0.7.3 (2025-07-17)

Features

• enable filtering for recombinant lineages with specifiable mode for including all recombinant sublineages or only the ones that are fully contained in the clade of the filtered node (6e335dd)

Bug Fixes

• benchmarking: only depend on WisePulse for the tool to run, not for the data (#875) (1aa4dc9)
• silo: handle newick files with new line at end (#889) (7f31fa7)

0.7.2 (2025-07-08)

Features

• add the field additionalFields to Fasta action (#861) (80777f1)
• benchmarking: add benchmarking/ directory with benchmark runner code (be93833)
• benchmarking: add copy of the evobench-probes library (b73e414)
• benchmarking: add probe on query plan execution (f2d96df)
• benchmarking: add probes on all operator evaluate methods (96b14c6)
• benchmarking: add probes on Date sort optimization (44b61de)
• benchmarking: re-use the TCP address/port immediately (8d8940f)
• enable Backpressure when streaming batches (3af7d04)
• show the error message when failing with a Poco::Net::NetException (07c42b2)
• silo: add MostRecentCommonAncestor action (#834) (157f186)

Bug Fixes

• benchmarking: update location of sorted_input_file.* (02daf40)
• better error message if the DateBetween column is not of type date (9b7fa99)
• change default streaming batch size from 50000 to 32767 to avoid reslicing batches (718fd22)
• improve error messages when ndjson file is invalid (#850) (dcff2e9)
• stop streaming response when network stream is interrupted (d58d668)
• use arrow::acero::SourceNode when constructing a TableScan (7035f1f)

0.7.1 (2025-07-02)

Bug Fixes

• add proper error messages for invalid groupByFields (2c8babb)
• evaluate the bitmaps in the preparation of the QueryPlan to not send wrong http headers (9c831c9)
• make TableScan react to StopProducing calls (#855) (ea3ee7d)
• throw an error when silo tries to start on a port that is already in use (e0ee23b)

0.7.0 (2025-06-30)

⚠ BREAKING CHANGES

• the field sequenceName in the Fasta and FastaAligned is now named sequenceNames and required to be an array (#804)
• change field sequenceName to sequenceNames in Insertions action (#821)
• change field sequenceName to sequenceNames in Mutations action (#819)
• rename DatabaseInfo field totalSize to verticalBitmapsSize and nBitmapsSize to horizontalBitmapsSize (#805)
• Null values are no longer ordered as the smallest element, but at the start or end instead. This is decided based on the sort-order of the first orderByField to mimic old behavior. This leads to a breaking change when ordering by multiple fields with mismatching order direction (#799)
• all unaligned nucleotide sequences are prefixed with unaligned_ in the api requests and response (#795)

Features

• streaming implementation of the SILO query engine based on the arrow framework (#765, #775, #792, #795, #799, #809, #821, #829, #837)
• cache DatabaseInfo, add amino acid sequence columns to size info (bc0d0a2)
• silo: add function to get all clades that are descendants of an internal node (#815) (d1f9989)
• silo: add recommended vscode extensions to .vscode folder for developers using vscode (d2c74db)
• silo: improve the docs and update build_with_conan.py (#779) (f176fd6)
• silo: read in phylogenetic tree files and create tree structure (#806) (f3d3f36)
• silo: refactor phylotree to use nodeId for children and parent and serialize (#822) (29e8361)

Bug Fixes

• avoid dropping the input bitmap of Selections in a nested And query (e87093c)
• correctly error when the input data contains illegal insertion characters (4867c32)
• remove erroneous warning printed on every preprocessing run (82b65c8)

0.6.0 (2025-04-23)

⚠ BREAKING CHANGES

• the DatabaseConfig fields dateToSortBy and partitionBy have been deprecated and do no longer have any effect. The internal endpoint for more detailed database storage information has been removed.

Features

• add append as a third silo execution mode for incremental preprocessing (84c4ae3), closes #368
• introduce a database schema that actually describes the schema of the database (be71e08)
• still enable silo preprocessing as a shorthand for initialize-then-append (9a91d0e)

0.5.9 (2025-04-16)

Bug Fixes

• return 503 response when no database has been loaded yet (#742) (79ebbaa), closes #723

0.5.8 (2025-03-31)

Bug Fixes

• the InsertionContains filter regex should match for the full string (#735) (4bb8320)

0.5.7 (2025-03-17)

Bug Fixes

• correctly expect an escaped stop codon in requests (9b3c27a)

0.5.6 (2025-02-27)

Features

• add coverage field to Mutations output and make it possible to only return subset of the fields (2b11472)

0.5.5 (2025-02-25)

Bug Fixes

• return proper error message when index is out of bounds for a HasMutations filter (50e9f49)

0.5.4 (2025-02-20)

Features

• add monitoring to log the swapover to datasets (39f1ed2)
• call malloc_trim after all connections to the old database are finished (5742a52)
• tell malloc to give empty heap space back to the OS before every fasta call (16ce850)

Bug Fixes

• prohibit allocation of new Poco threads for incoming requests (7c2994f)

0.5.3 (2025-02-06)

Features

• add endpoint that returns the lineageDefinitions for the requested column (#689) (69cd9b0)

0.5.2 (2025-01-29)

Features

• add the version to the output of the info endpoint (#680) (87e3f60), closes #604

0.5.1 (2025-01-28)

Features

• update base image to ubuntu:24.04 (46d0567)

Bug Fixes

• return an error if reserved value INT32_MIN is used in filter (042646b)

0.5.0 (2024-12-19)

⚠ BREAKING CHANGES

• new command line / config system

Features

• add materializationCutoff option (8920d5f)
• new command line / config system (2a29074), closes #633

Bug Fixes

• declare virtual constructor for abstract base class CustomSqlFunction (603ddf6)

0.4.0 (2024-11-14)

⚠ BREAKING CHANGES

• metadataInputFile key in preprocessing config file has been removed. Instead, ndjson files should be used and specified with the ndjsonInputFilename option

Features

• remove tsv/fasta/sam input format (#562) (fd7dc6f)

Bug Fixes

• preprocessing: correct error message when field is in config but not in metadata file (828182a)
• remove shared_ptr in lambda captures which might lead to memory leaks (a26d8b8)

0.3.2 (2024-10-30)

Bug Fixes

• preprocessing: add column_name to column classes and add them to error messages (e011e5c) (#629)

0.3.1 (2024-10-25)

Bug Fixes

• preprocessing: error on duplicate primary keys (687c3ef)
• validate the column types earlier to prevent duckdb auto-casting to cast types with differing round-trip conversions (ac238b2)

0.3.0 (2024-10-16)

⚠ BREAKING CHANGES

• generalized wildcard queries (#458)
  • The preprocessing config field pangoLineageDefinitionFilename has been renamed to lineageDefinitionFilename.
  • We now accept a YAML lineage definition file instead of a Pango alias key.
  • Input and query validation now checks whether the provided lineage exists in the defined lineages, and errors are thrown if validation fails.
  • The metadata type pangoLineage has been removed. type: string with generateLineageIndex: true should be used instead.

Features

• generalized wildcard queries (#458) (1432320)

Bug Fixes

• resolve aliases when inserting to or querying lineage indexes again (0e841e6)
• update script to also generate aliases (43ac4aa)

0.2.24 (2024-10-15)

Bug Fixes

• prevent setDatabase starvation which leads to no updates of currently loaded database (24e4db6) (#613)

0.2.23 (2024-10-10)

Bug Fixes

• preprocessing: enforce exact match on file base stem when pairing sequence and metadata when processing from file (7c9b23d), closes #607

0.2.22 (2024-10-01)

Bug Fixes

• correctly escape quotes in field names (#595) (7e7b448)

0.2.21 (2024-09-20)

Bug Fixes

• preprocessing: resolves spurious OOM crashes when handling large datasets (6e4eae2)

0.2.20 (2024-09-17)

Features

• make duckdb memory limit configurable via preprocessing config (c112eb1)

0.2.19 (2024-09-12)

Bug Fixes

• preprocessing: validate that either a ndjson or tsv file was given as input (#564) (7bf2ef7)

0.2.18 (2024-09-12)

Features

• do not abort on assertment failures (4e5f598)
• panic: add ASSERT, UNREACHABLE, UNIMPLEMENTED, safer env var (867d08f)
• update duckdb version (577c1b2)

Bug Fixes

• duplicate key in sample.ndjson (c9b5232)
• increase duckdb memory limit to 80GB (20131bb)
• order of randomized tests changed (d29b2f2)

0.2.17 (2024-08-30)

Bug Fixes

• manually overwrite already existing unaligned sequence files in the temp folder (f37fd88)

  • In case the unaligned sequence directory already existed in the temp folder when running preprocessing, it is manually deleted. Before, this lead to a bug, as the currently used duckdb version does not correctly overwrite the temp files and may leave duplicate entries in the output. This is fixed in duckdb PR #11787 which is included in release v0.10.3

0.2.16 (2024-08-29)

Bug Fixes

• do not order keys and sequences differently, leading to wrong unaligned sequence results when dateToSortBy is set (b0f8352)
• do not return incorrect results when encountering a null value in the unaligned sequence action (a69d117)
• fix the RIGHT JOIN in preprocessing of unaligned sequences to contain the correct primary key (a25d5d0)

0.2.15 (2024-08-23)

Bug Fixes

• don't abort when unaligned sequence is missing - we should return null (#542) (2c649cb), closes #541

0.2.14 (2024-08-19)

Features

• new operator for matching strings against regular expressions (ade25e8)

Bug Fixes

• typo in the error message if a column is not found (67029a4)

0.2.13 (2024-08-06)

Features

• bump serialization version to 2 (e897203)
• bump serialization version to 3 (7584214)
• support SAM files as sequence input and allow partial sequence input with an offset (5be9a9f)

Bug Fixes

• allow all sequence-names by escaping them properly in all SQL statements (901fc7e)

0.2.12 (2024-07-31)

Features

• change base image to ubuntu (64dee0d)

0.2.11 (2024-07-24)

Bug Fixes

• streaming: report an error when sorting requested while streaming (313d6bb)

0.2.10 (2024-07-22)

Bug Fixes

• reduce function calls and parallelism when inserting rows into storage backend (#509) (93ba858)
• preprocessing: don't abort inserting insertions upon finding a null value (03a500a)

0.2.9 (2024-07-18)

Features

• have no default sequence by default, implement default amino acid sequence from config (2edf924), closes #454
• set the default sequence when there is only a single sequence in the reference genomes (0ae4022), closes #454
• throw error in preprocessing when default sequence is not in reference genomes (287bb8b), closes #454

Bug Fixes

• streaming: report an error when the unimplemented offset or limit matters (3be9c06), closes #511

0.2.8 (2024-07-08)

Features

• the Fasta and FastaAligned actions now stream their result instead of allocating it in memory, closes #112

0.2.7 (2024-06-25)

Bug Fixes

• don't abort when reading table in chunks (26b1558)

0.2.6 (2024-06-19)

Bug Fixes

• trigger release due to refactor commit-5e3041e5 (#494) (#496) (ecd4ba0)

0.2.5 (2024-06-17)

Bug Fixes

• empty input without partitioning (5fa3c92)

0.2.4 (2024-06-14)

Features

• allow null for sequenceName in insertion contains queries (6dbe251)

Bug Fixes

• more efficient ndjson emptiness check (#481) (344ec7b)

0.2.3 (2024-06-10)

Bug Fixes

• be able to start without genes, without nucleotide sequences or with neither (e878ed5)
• empty ndjson input files more robust (#473) (9d4232b)
• erroneous file created during unit tests should not leak (1b764af)
• insertions being added at wrong index for large files (#472) (e056ed9)
• remove incorrect compile flags (b92ee4e)

0.2.2 (2024-05-31)

Bug Fixes

• start with empty files without throwing an error (d407b92)

0.2.1 (2024-05-28)

Bug Fixes

• allow insertions that start at 0 (#449) (f427137)

0.2.0 (2024-05-23)

⚠ BREAKING CHANGES

• old database_config files might be invalid if they contained insertion columns. Also, we are more prohibitive for ndjson input files, which now MUST contain nucleotide/amino acid insertions for all respective sequences. The insertions action and filter do no longer require a column field.

Features

• insertions no longer in metadata or databaseConfig, instead expected for all aligned sequences #372 (1f3680c)

0.1.1 (2024-05-23)

Features

• add number of partitions to /info (1e780b5)
• add serializationVersion to SILO output for smoother transitioning to new formats (d3badb6)

0.1.0 (2024-05-10)

⚠ BREAKING CHANGES

• return data as NDJSON instead of JSON

Features

• AAMutations with multiple sequences (0def8b2)
• Action for amino acid distribution (a0a4cf1)
• add limit orderBy and offset to all query actions (13b7e01)
• add log statements to loadDatabaseState (46a0421)
• add more tests, make less flaky and viable with large dataset (7772ae3)
• add unit test for findIllegalNucleotideChar, unique test case name for insertion contains invalid pattern tests (99c9c4b)
• added amino acid insertion search, added many test cases and fixed various bugs (d1e4b2b)
• allow default preprocessing config along with user defined preprocessing config (ee9f20e)
• allow reading fasta files with missing segments and genes #220 (8ea9893)
• allow reading segments and genes that are null from ndjson file #220 (d0a3a7e)
• also get Runtime Config options from environment variables (33bdd65)
• also log to stdout (54b8a47)
• also return mutation destructed that does not need to be reparsed (a93abbf)
• Alternative templating of symbol classes (6b61985)
• automatically detect file endings for fasta files (75bd14e)
• be more lenient on input data, ignore superfluous sequences and fill missing sequences with Ns (ee12186)
• Better test coverage for SymbolEquals filter (42c685c)
• boolean columns, resolves #384: const declaration (685db9f)
• boolean columns: actions/tuple: update assignTupleField() (b020109)
• boolean columns: add and use JsonValueType (2ad1268)
• boolean columns: add bool to JsonValueType, update tuple (3c990e2)
• boolean columns: add expression_type "BooleanEquals" (b82ec69)
• boolean columns: add filter_expressions/bool_equals (ff2a138)
• boolean columns: add optional_bool (a2e47f8)
• boolean columns: add storage/column/bool_column (5eb3430)
• boolean columns: column_group: update ColumnPartitionGroup (d7adecf)
• boolean columns: column_group.h: add {ColumnPartitionGroup,ColumnGroup}::bool_columns fields (af44f1a)
• boolean columns: database (c555a68)
• boolean columns: database_config: add "bool" case to DatabaseConfigReader::readConfig() (12fc7b8)
• boolean columns: database_config: add "boolean" case to de/serialisation (e6d5363)
• boolean columns: database_config: add BOOL to ValueType (0b47b82)
• boolean columns: database_config: update DatabaseMetadata::getColumnType() (30febdd)
• boolean columns: database_partition (0aef8f0)
• boolean columns: optional_bool: add == (f0aa3e8)
• boolean columns: selection (e52f2dc)
• build metadata in parallel to sequences. Do not create unaligned sequence tables in preprocessing, rather hive-partition them directly to disk. Better (debug-)logging (c1cdfeb)
• bulk Tuple allocations now possible (902ec04)
• clearer Operator::negate and Expression::toString, logical Equivalents for debug printing/logging for the Leaf Operators IndexScan and BitmapSelection (026b639)
• consistent behavior of configs when starting SILO with both --preprocessing and --api (847ec7e)
• declutter README.md from linting option, which is now disabled by default and enforced in the CI for the Linter (9220435)
• details no longer shows insertions (#354) (473cd98)
• display database info after loading new database state (0249416)
• display preprocessing duration in logs in human-readable format (not in microseconds) #296 (a2499af)
• do not enforce building with clang-tidy by default. Linter will still be enforced (7134e45)
• FastaAligned action (50776c8)
• faster builds by copying @corneliusroemer image caching for our dependency images, which rarely change (#374) (7867bc7)
• filter for amino acids (b52aabd)
• fix sorting (1ed18ae)
• flipped bitmap can now be set before insertion (f61c803)
• format DatabaseConfig (4fb8f1b)
• format PreprocessingConfig (ee35207)
• generalize mutations action to have consistent behavior for different symbols (9834aea)
• generalizing symbol and mutation filters. Clear handling of ambiguous symbols (aa9ad4d)
• Generalizing the config for multiple nucleotide sequences and multiple genes (9a80204)
• have structured and destructured insertion in insertions response (0a7e46a)
• hide intermediate results of the preprocessing - don't put it in the output (44327b0)
• implement basic request id to trace requests #303 (4defb59)
• implement data updates at runtime. More resilient to superfluous or missing directory separators (dc5dfaa)
• implement insertion columns and search (9167236)
• improve loadDB speeds (2b7cd7d)
• improve validation error message of some actions on orderByFields (a0da5b5)
• insertion action targets all insertion columns by default (6b70241)
• insertion columns for amino acids and multiple sequence names (3cc8fee)
• insertions action (e067062)
• insertions contains action now targets all columns if the column name is missing (32a6951)
• introduce new storage type for Sequence Positions, where the most numerous symbol is deleted (6e15204)
• introduce storage of unaligned sequences from either ndjson file or fasta file and make them queryable via the Fasta action (44df849)
• load table lazily. Unaligned Sequences do not need to load the table (c2a8439)
• log databaseConfig and preprocessingConfig (d2dc58c)
• logging for partition (e75a925)
• logging improvements (4c12a88)
• make database serializable again (2523e67)
• make pangoLineageDefinitionFilename in preprocessing config optional, linter errors (0f3dc53)
• make partition_by field in config optional (3942418)
• make SILO Docker image by default read data from /data (e83b910)
• make threads and max queued http connections available through optional parameter (3ecde68)
• migration to duckdb 0.10.1 (c1426ef)
• mine data version at beginning of preprocessing (362fe0f)
• More robust InputStreamWrapper (305dd36)
• multiple performance improvements for details endpoint (28f41d0)
• optimize bitmaps before finishing partition (5b06d58)
• order all actions by default (a2f5c04)
• preparation of insertion columns (c14a370)
• put output and logs to gitignore (789e489)
• reenable bitmap inversion (75ac20f)
• reenable pushdown of And expressions through selections (802bec0)
• refactor saving and loading database to not require preprocessing structs anymore (45bf7ed)
• reintroduce randomize for all query actions (166045c)
• reserve space in columns when bulk inserting rows (e3c9620)
• return data as NDJSON instead of JSON (c236ba4), closes #126
• return data version on each query (be5c886)
• return only aliased pango lineages (abf0844)
• reveal some more details when reading YAML fails (1f8d9db)
• run preprocessing in github ci (f53eddb)
• save database state into folder with name <data version> (41923eb)
• separate preprocessing and starting silo (9808e2d)
• serialization of partition descriptor to json (472c1da)
• some suggestions for the insertion search (ae900da)
• Specifiable nucSequence query target (7cc609f)
• statically disable deleted symbol optimisations because of performance penalty (4e522f4)
• stick to the default of having config value keys in camel case (a1cae40)
• storing amino acids (f11a330)
• support for nullable columns (6f78e3a)
• support recombinant lineages (3e848a5)
• template class for sequence store (cef4d48)
• templatized Symbol classes (6b9d734)
• test set with amino acid insertions (de2c4f8)
• throw an error when there is not initialized database loaded yet #295 (b17f72a)
• tidying up CLI and file configuration for runtime config. Added option for specifying the port (c3a88a0)
• Unit tests for Tuple (4fc06e8)
• update conan version (5540a67)
• use 'pragma once' as include guards instead of 'ifndef...' (bc49aa5)
• use own scope for preprocessing (2a93846)
• use same default min proportions for mutations actions as the old LAPIS (f42f830)

Bug Fixes

• adapt randomize query results to target architecture. x86 and ARM have possibly different std::hash results (#355) (600000d)
• add bash dependency which is required by conan build of pkgconf and is not installed on alpine by default (1b3f51c)
• add insertion to database_config test (aec40d0)
• add missing file for test (262bedc)
• add missing sequenceName field to mutation action "orderBy" (06c8c86)
• add sleep statement before row call (8fa8efb)
• add workaround so insertions are read correctly (8a2bfa8)
• allow sql keywords for metadata field names #259 (6fbeee5)
• also consider 'missing' symbols in the mutation action. Bugfix where Position invariant was broken because of 'missing' symbols (fab72a6)
• alternative non-exhaustive three mer index (467086f)
• always build dependency image for amd64 platform (4f73cb0)
• bug when filtering for indexedStringColumns which are not present in some partitions (62d4e08)
• bug where sequence reconstruction is false when the flipped bitmap is different from the reference sequence symbol (edac58c)
• change random ordering for gcc hashing (78055ff)
• change test to reflect new optimisations (cb63010)
• compiling And: append selection_child->predicates to predicates - not vice versa (bdebca5)
• deterministic order for e2e test (02427a2)
• divergence between mac and linux info test results, fix memory leaks in Threshold.cpp (ffcefd0)
• do not exclude zstd filter from boost installation (c974230)
• do not use std::filesystem::path::relative_path() to also support absolute paths (b9ff422)
• endToEndResults (c807a33)
• error when the Mutations action looked for sequences but the filter was empty (91e5d52)
• fix memory leaks in indexed_string_column.cpp and insertion_contains.cpp (c2eeac8)
• floatEquals and floatBetween with null values (47b436e)
• hide nucleotide sequence for default sequence (584715c)
• insertion column, remove reference (d940c52)
• insertion search e2e and insertion column tests, dont allow non-empty value for insertion search (19af04d)
• linking error on linux (ad9076c)
• linter (54d3c34)
• linter (e6b6ab7)
• linter (b31851c)
• linter (34830ab)
• linter errors (e9e1bbf)
• Linter throws again and added clang-format option (87cb4a5)
• Make C++ flags in CMake compatible for MacOS (38cae69)
• metadata info test accessing getMetadataFields output no longer directly but over the address of a const (0dcdee1)
• missing include (0773999)
• new linter errors (ea6934c)
• no longer have regression when no bitmap flipped is most efficient (b653753)
• nodiscard (silence warnings) (69421c1), closes #390
• non default unaligned nucleotide sequence prefix (93b4829)
• nucleotide symbol equals with dot (6ad623e)
• only apply order-by if the field is set, validate orderBy fields for all operations (405a7f1)
• pango lineage filter with null values (b7238a8)
• parse error messages for mutation filter expressions (9e4612d)
• put compressors into sql function to avoid static variables (c1a11c8)
• quoting {} in "x.{}" SQL struct accesses, as a string starting with a number leads to parser errors (#409) (f3ba6db)
• random (but deterministic for a version) result can depend on internal state, which was changed with duckdb update (9d9351b)
• recursive file reading for nodejs<20 (9ac0ffe)
• remove caching for linter. Docker image to large on github actions. (8eaeee6)
• response format (bca4961)
• revert duckdb migration due to it being unable to build the new version on the GitHub runner (2a62c54)
• revert test numbers to pre-optimization (99b600a)
• Roaring from 1.3.0 -> 1.0.0 because of broken CI (ce82c77)
• seralization for insertion_index and insertion_column (ee99f8d)
• single partition build fix (a8af1c9)
• specify namespace fmt in calls to format_to (#353) (62ffa3b)
• specifying apk versions (2c2354e)
• test cases verifying that the positions index for mutation distribution are now 1-indexed (fdf972a)
• test with deterministic results, remove 2 unused variables (96a424b)
• unit test info number updates for new pango-lineages in test data (39d07c2)
• unit tests and mock fixtures (db506a1)
• update cmake version on ubuntu (227a2dd)
• Upper and lower bound should be inclusive in DateBetween filter (53c6c05)
• Wrong compare function used in multi-threaded case, which displayed wrong tuples in the details endpoint when a limit was used (650bf36)
• zstd dependency (c145722)