Skip to content

770 migrate insertions action - #821

Merged
taepper merged 4 commits into
mainfrom
770-Migrate-Insertions-action
Jun 25, 2025
Merged

770 migrate insertions action#821
taepper merged 4 commits into
mainfrom
770-Migrate-Insertions-action

Conversation

@taepper

@taepper taepper commented Jun 23, 2025

Copy link
Copy Markdown
Collaborator

resolves #770

Summary

This was very straight-forward with the same approach as for Mutations

PR Checklist

- [ ] All necessary documentation has been adapted or there is an issue to do so.
- [ ] The implemented feature is covered by an appropriate test.

@github-actions

github-actions Bot commented Jun 23, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

This is a preview of the changelog of the next release. If this branch is not up-to-date with the current main branch, the changelog may not be accurate. Rebase your branch on the main branch to get the most accurate changelog.

Note that this might contain changes that are on main, but not yet released.

Changelog:

0.7.0 (2025-06-25)

⚠ BREAKING CHANGES

  • change field sequenceName to sequenceNames in Insertions action
  • change field sequenceName to sequenceNames in Mutations action
  • rename DatabaseInfo field totalSize to verticalBitmapsSize and nBitmapsSize to horizontalBitmapsSize
  • the field sequenceName in the Fasta and FastaAligned is now named sequenceNames and required to be an array
  • streaming implementation of Details action. select-k also for Fasta and FastaAligned
  • streaming implementation of the Fasta action with the arrow framework

Features

  • arrow framework for query engine and specialized implementation for FastaAligned action (b719e15), closes #654
  • cache DatabaseInfo, add amino acid sequence columns to size info (bc0d0a2)
  • silo: add recommended vscode extensions to .vscode folder for developers using vscode (d2c74db)
  • silo: improve the docs and update build_with_conan.py (#779) (f176fd6)
  • silo: read in phylogenetic tree files and create tree structure (#806) (f3d3f36)
  • streaming implementation of the Fasta action with the arrow framework (0392207)
  • update onetbb, nlohmann_json, gtest, roaring, yaml-cpp and zstd to most recent versions (3abd2d9)

Bug Fixes

  • bug in table_scan, that confused row_id in the database with row_id in the current batch (9b7dce4)
  • correctly error when the input data contains illegal insertion characters (4867c32)
  • make better use of the arrow::acero API, no longer rely on passing the stream inside an extra operator (dabb186)
  • missing initialization of uint32_t in config struct, set default value in ConfigSpecification instead (#812) (baf0fcc)
  • properly catch exceptions during arrow execution to avoid program termination (6caee0e)
  • remove erroneous warning printed on every preprocessing run (82b65c8)

Code Refactoring

  • change field sequenceName to sequenceNames in Insertions action (dfb7ac7)
  • change field sequenceName to sequenceNames in Mutations action (851b5a9)
  • make sequenceName in Fasta and FastaAligned actions an array (6c345fb)
  • streaming implementation of Details action. select-k also for Fasta and FastaAligned (bfbb3ee)

@taepper
taepper force-pushed the 770-Migrate-Insertions-action branch from 4e4159d to 05fd894 Compare June 23, 2025 10:07
fengelniederhammer
fengelniederhammer reviewed Jun 24, 2025
View reviewed changes

@fengelniederhammer fengelniederhammer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but I think it would be good to add a unit test since we don't have one (or did I miss it?)

@taepper
taepper force-pushed the 770-Migrate-Insertions-action branch from 05fd894 to ff66890 Compare June 24, 2025 15:06
fengelniederhammer
fengelniederhammer reviewed Jun 25, 2025
View reviewed changes

@fengelniederhammer fengelniederhammer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, the tests are still failing though.

@taepper
taepper force-pushed the 770-Migrate-Insertions-action branch from ff66890 to 8cc8405 Compare June 25, 2025 08:07
@taepper
taepper merged commit 285ab66 into main Jun 25, 2025
9 checks passed
@taepper
taepper deleted the 770-Migrate-Insertions-action branch June 25, 2025 08:48
@chaoran-chen chaoran-chen mentioned this pull request Jul 18, 2025
Merged
chaoran-chen added a commit to loculus-project/loculus that referenced this pull request Jul 23, 2025
@chaoran-chen
feat(kubernetes): update SILO to 0.7.4 and LAPIS to 0.5.8 (#4717)
bc8ee54 
# SILO Change Log

##
[0.7.4](GenSpectrum/LAPIS-SILO@v0.7.3...v0.7.4)
(2025-07-21)


### Bug Fixes

* **silo:** accept more characters in newick string and improve error
messages ([#891](GenSpectrum/LAPIS-SILO#891))
([5f8bc9e](GenSpectrum/LAPIS-SILO@5f8bc9e))
* **silo:** correctly deduplicate requested fields
([78f4146](GenSpectrum/LAPIS-SILO@78f4146))

##
[0.7.3](GenSpectrum/LAPIS-SILO@v0.7.2...v0.7.3)
(2025-07-17)


### Features

* enable filtering for recombinant lineages with specifiable mode for
including all recombinant sublineages or only the ones that are fully
contained in the clade of the filtered node
([6e335dd](GenSpectrum/LAPIS-SILO@6e335dd))


### Bug Fixes

* **benchmarking:** only depend on WisePulse for the tool to run, not
for the data
([#875](GenSpectrum/LAPIS-SILO#875))
([1aa4dc9](GenSpectrum/LAPIS-SILO@1aa4dc9))
* **silo:** handle newick files with new line at end
([#889](GenSpectrum/LAPIS-SILO#889))
([7f31fa7](GenSpectrum/LAPIS-SILO@7f31fa7))

##
[0.7.2](GenSpectrum/LAPIS-SILO@v0.7.1...v0.7.2)
(2025-07-08)


### Features

* add the field `additionalFields` to Fasta action
([#861](GenSpectrum/LAPIS-SILO#861))
([80777f1](GenSpectrum/LAPIS-SILO@80777f1))
* **benchmarking:** add benchmarking/ directory with benchmark runner
code
([be93833](GenSpectrum/LAPIS-SILO@be93833))
* **benchmarking:** add copy of the evobench-probes library
([b73e414](GenSpectrum/LAPIS-SILO@b73e414))
* **benchmarking:** add probe on query plan execution
([f2d96df](GenSpectrum/LAPIS-SILO@f2d96df))
* **benchmarking:** add probes on all operator evaluate methods
([96b14c6](GenSpectrum/LAPIS-SILO@96b14c6))
* **benchmarking:** add probes on Date sort optimization
([44b61de](GenSpectrum/LAPIS-SILO@44b61de))
* **benchmarking:** re-use the TCP address/port immediately
([8d8940f](GenSpectrum/LAPIS-SILO@8d8940f))
* enable Backpressure when streaming batches
([3af7d04](GenSpectrum/LAPIS-SILO@3af7d04))
* show the error message when failing with a Poco::Net::NetException
([07c42b2](GenSpectrum/LAPIS-SILO@07c42b2))
* **silo:** add MostRecentCommonAncestor action
([#834](GenSpectrum/LAPIS-SILO#834))
([157f186](GenSpectrum/LAPIS-SILO@157f186))


### Bug Fixes

* **benchmarking:** update location of sorted_input_file.*
([02daf40](GenSpectrum/LAPIS-SILO@02daf40))
* better error message if the DateBetween column is not of type date
([9b7fa99](GenSpectrum/LAPIS-SILO@9b7fa99))
* change default streaming batch size from 50000 to 32767 to avoid
reslicing batches
([718fd22](GenSpectrum/LAPIS-SILO@718fd22))
* improve error messages when ndjson file is invalid
([#850](GenSpectrum/LAPIS-SILO#850))
([dcff2e9](GenSpectrum/LAPIS-SILO@dcff2e9))
* stop streaming response when network stream is interrupted
([d58d668](GenSpectrum/LAPIS-SILO@d58d668))
* use arrow::acero::SourceNode when constructing a TableScan
([7035f1f](GenSpectrum/LAPIS-SILO@7035f1f))

##
[0.7.1](GenSpectrum/LAPIS-SILO@v0.7.0...v0.7.1)
(2025-07-02)


### Bug Fixes

* add proper error messages for invalid groupByFields
([2c8babb](GenSpectrum/LAPIS-SILO@2c8babb))
* evaluate the bitmaps in the preparation of the QueryPlan to not send
wrong http headers
([9c831c9](GenSpectrum/LAPIS-SILO@9c831c9))
* make TableScan react to StopProducing calls
([#855](GenSpectrum/LAPIS-SILO#855))
([ea3ee7d](GenSpectrum/LAPIS-SILO@ea3ee7d))
* throw an error when silo tries to start on a port that is already in
use
([e0ee23b](GenSpectrum/LAPIS-SILO@e0ee23b))

##
[0.7.0](GenSpectrum/LAPIS-SILO@v0.6.0...v0.7.0)
(2025-06-30)


### ⚠ BREAKING CHANGES

* the field sequenceName in the Fasta and FastaAligned is now named
sequenceNames and required to be an array
([#804](GenSpectrum/LAPIS-SILO#804))
* change field sequenceName to sequenceNames in Insertions action
([#821](GenSpectrum/LAPIS-SILO#821))
* change field sequenceName to sequenceNames in Mutations action
([#819](GenSpectrum/LAPIS-SILO#819))
* rename DatabaseInfo field `totalSize` to `verticalBitmapsSize` and
`nBitmapsSize` to `horizontalBitmapsSize`
([#805](GenSpectrum/LAPIS-SILO#805))
* Null values are no longer ordered as the smallest element, but at the
start or end instead. This is decided based on the sort-order of the
first orderByField to mimic old behavior. This leads to a breaking
change when ordering by multiple fields with mismatching order direction
([#799](GenSpectrum/LAPIS-SILO#799))
* all unaligned nucleotide sequences are prefixed with `unaligned_` in
the api requests and response
([#795](GenSpectrum/LAPIS-SILO#795))

### Features

* streaming implementation of the SILO query engine based on the arrow
framework ([#765](GenSpectrum/LAPIS-SILO#765),
[#775](GenSpectrum/LAPIS-SILO#775),
[#792](GenSpectrum/LAPIS-SILO#792),
[#795](GenSpectrum/LAPIS-SILO#795),
[#799](GenSpectrum/LAPIS-SILO#799),
[#809](GenSpectrum/LAPIS-SILO#809),
[#821](GenSpectrum/LAPIS-SILO#821),
[#829](GenSpectrum/LAPIS-SILO#829),
[#837](GenSpectrum/LAPIS-SILO#837))
* cache DatabaseInfo, add amino acid sequence columns to size info
([bc0d0a2](GenSpectrum/LAPIS-SILO@bc0d0a2))
* **silo:** add function to get all clades that are descendants of an
internal node
([#815](GenSpectrum/LAPIS-SILO#815))
([d1f9989](GenSpectrum/LAPIS-SILO@d1f9989))
* **silo:** add recommended vscode extensions to .vscode folder for
developers using vscode
([d2c74db](GenSpectrum/LAPIS-SILO@d2c74db))
* **silo:** improve the docs and update build_with_conan.py
([#779](GenSpectrum/LAPIS-SILO#779))
([f176fd6](GenSpectrum/LAPIS-SILO@f176fd6))
* **silo:** read in phylogenetic tree files and create tree structure
([#806](GenSpectrum/LAPIS-SILO#806))
([f3d3f36](GenSpectrum/LAPIS-SILO@f3d3f36))
* **silo:** refactor phylotree to use nodeId for children and parent and
serialize ([#822](GenSpectrum/LAPIS-SILO#822))
([29e8361](GenSpectrum/LAPIS-SILO@29e8361))


### Bug Fixes

* avoid dropping the input bitmap of Selections in a nested And query
([e87093c](GenSpectrum/LAPIS-SILO@e87093c))
* correctly error when the input data contains illegal insertion
characters
([4867c32](GenSpectrum/LAPIS-SILO@4867c32))
* remove erroneous warning printed on every preprocessing run
([82b65c8](GenSpectrum/LAPIS-SILO@82b65c8))

# LAPIS Change Log

## [0.5.8](GenSpectrum/LAPIS@v0.5.7...v0.5.8)
(2025-07-21)


### Features

* **lapis:** add aminoAcidMutationsOverTime
([#1270](GenSpectrum/LAPIS#1270))
([c9adf26](GenSpectrum/LAPIS@c9adf26)),
closes [#1214](GenSpectrum/LAPIS#1214)


## [0.5.7](GenSpectrum/LAPIS@v0.5.6...v0.5.7)
(2025-07-18)


### Features

* **lapis:** add parameter `fastaHeaderTemplate` to customize the fasta
header in sequences endpoints
([#1258](GenSpectrum/LAPIS#1258))
([ac7fc29](GenSpectrum/LAPIS@ac7fc29)),
closes [#857](GenSpectrum/LAPIS#857)
* **lapis:** drop sorting by segment/gene from docs of sequence
endpoints ([#1268](GenSpectrum/LAPIS#1268))
([bc67a7c](GenSpectrum/LAPIS@bc67a7c)),
closes [#1248](GenSpectrum/LAPIS#1248)

## [0.5.6](GenSpectrum/LAPIS@v0.5.5...v0.5.6)
(2025-07-15)


### Features

* **lapis:** add nucleotideMutationsOverTime endpoint
([#1229](GenSpectrum/LAPIS#1229))
([cade8d9](GenSpectrum/LAPIS@cade8d9)),
closes [#1205](GenSpectrum/LAPIS#1205)

## [0.5.5](GenSpectrum/LAPIS@v0.5.4...v0.5.5)
(2025-07-01)


### Bug Fixes

* **lapis:** also return sequences that are null when requesting JSON or
NDJSON ([#1256](GenSpectrum/LAPIS#1256))
([a888965](GenSpectrum/LAPIS@a888965)),
closes [#1255](GenSpectrum/LAPIS#1255)

## [0.5.4](GenSpectrum/LAPIS@v0.5.3...v0.5.4)
(2025-07-01)


### Features

* **lapis:** adapt to SILO changes that renamed the unaligned sequences
to have the "unaligned_" prefix
([#1225](GenSpectrum/LAPIS#1225))
([d669335](GenSpectrum/LAPIS@d669335))
* **lapis:** add endpoint `/sample/alignedAminoAcidSequences` to
download all genes at once
([#1242](GenSpectrum/LAPIS#1242))
([36dfd5f](GenSpectrum/LAPIS@36dfd5f))
* **lapis:** add endpoint `/sample/unalignedNucleotideSequences` for
multi-segmented organism to download several sequences at once
([#1246](GenSpectrum/LAPIS#1246))
([f9275c2](GenSpectrum/LAPIS@f9275c2))
* **lapis:** add endpoint to download all aligned segments (or a subset)
at once (for multi segmented organisms)
([#1235](GenSpectrum/LAPIS#1235))
([6207d32](GenSpectrum/LAPIS@6207d32))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fengelniederhammer fengelniederhammer fengelniederhammer approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Migrate Insertions action to new framework

2 participants

@taepper @fengelniederhammer