release: v0.9.2

Highlights

The main highlights of this release are:

• immudb gRPC API now support streams.
  The Go SDK provides the methods Stream* to perform streaming operations. Eg. StreamSet, StreamGet
• In the past, immudb returned by default the last indexed transaction. As the index was asynchronous, it may not be the last insertion, unless SinceTx was specified with a value bigger than the current transaction.
  immudb now defaults to a more intuitive behavior, where operations will wait until indexing is completed and return the last transaction, unless NoWait=true is specified.
• immudb now performs index compaction asynchronously, without blocking insertions.

Changelog

[v0.9.2] - 2021-04-07

Bug Fixes

• fix StreamVerifiedSet and Get and add an (integration) test for them
• fix inclusion proofs in StreamVerifiedSet and Get
• password reader 'inappropriate ioctl for device' from stdin (#658)
• include AtTx in StreamZScan response
• embedded: fix data races
• embedded: use mutex to sync ops at tx lru-cache
• embedded/store: sync ReadTx operation
• embedded/store: fix indexing sync and error retrieval
• embedded/store: ensure waitees get notified when store is restarted
• embedded/store: remove checking for closed store when fetching any vlog
• embedded/store: continue indexing once index is replaced with compacted index
• embedded/store: set delay with duration in ms
• embedded/store: ensure watchers get notified when indexing is up-to-date
• embedded/tbtree: release lock when compaction thld was not reached
• embedded/tbtree: insertion delay while compacting not affecting compaction
• embedded/tbtree: set lastSnapshot once flushed is completed
• pkg/auth: add missing stream write methods to permissions
• pkg/client: fix minor leftover
• pkg/client: fix security issue: if client local state became corrupted an error is returned
• pkg/client: ensure dual proof verification is made when there is a previously verified state
• pkg/database: wrap seekKey with prefix only when seekKey is non-empty
• pkg/server: use latest snapshot when listing users

Changes

• add max tx values length guard and remove code duplication
• move stream service to a proper package
• improve serverside stream error handling
• add video streamer command
• renaming stream methods
• updating copyright
• renaming stream methods, add stubs and stream service factory
• fix binary notation
• in server store creation max value entry is fixed to 32Mb
• mocked server uses the inner immudb grpc server and can be gracefully stopped
• set stream supports multiple key values
• fixed minimum chunk size at 4096 bytes
• refactor code quality issues
• increase stream coverage and add a guard if key is present on a stream but no value is found
• remove fake proveSinceTxBs key send in streamVerifiableSet
• polish streams methods and add comments
• embedded: remove unused cbuffer package
• embedded: add logger
• embedded: fix some typos with comments
• embedded: log indexing notifications
• embedded: descriptive logs on indexing and already closed errors
• embedded: add logs into relevant operations
• embedded: compaction and snapshot handling
• embedded/appendable: thread-safe multi-appendable
• embedded/appendable: sync before copying appendable content
• embedded/appendable: multi-appendable fine-grained locking
• embedded/store: remove duplicated logging
• embedded/store: stop indexing while commiting with callback
• embedded/store: leverage fine-grained locking when reading tx data
• embedded/store: general improvements on snapshot management
• embedded/store: log number of transactions yet to be indexed
• embedded/store: use buffered channel instead of a circular buffer
• embedded/store: set a limit on indexing iteration
• embedded/store: set max file size to 2Gb (#649)
• embedded/store: remove conditional locking before dumping index
• embedded/store: lock-less readTx
• embedded/tbtree: terminate reader if prefix won't match any more
• embedded/tbtree: optimize seek position
• embedded/tbtree: revert seek key setting
• embedded/tbtree: sync key-history log during compaction
• embedded/tbtree: sync before dumping
• embedded/tbtree: optimize seek position
• embedded/watchers: broadcasting optimisation
• embedded/watchers: minor renaming
• embedded/watchers: accept non-continuous notification
• pkg/client: add a guard to check for min chunk size
• pkg/client: add stream service factory on client and increase stream coverage
• pkg/client: maps server error on client package
• pkg/client: remove local files tests
• pkg/client: add GetKeyValuesFromFiles helper method
• pkg/client: integration test is skipped if immudb server is not present
• pkg/database: return error while waiting for index to be up to date
• pkg/database: use in-mem current snapshot in execAll operation
• pkg/database: leverage lightweight waiting features of embedded store
• pkg/database: return error while waiting for index to be up to date
• pkg/database: ensure scan runs over fully up-to-date snapshot
• pkg/database: illegal state guard is added to verifiable get and getTx methods
• pkg/server: max recv msg size is set to 32M
• pkg/server: add a guard to check for min chunk size
• pkg/server: revert quit chan exposure
• pkg/server: add server error mapper interceptor
• pkg/server: add small delay for indexing to be completed
• pkg/server: exposes Quit chan
• pkg/stream: remove bufio.reader when not needed
• pkg/stream: remove duplicated code
• pkg/stream: add ErrNotEnoughDataOnStream error and ImmuServiceReceiver_StreamMock
• pkg/stream: renamed stream test package
• pkg/stream: add a guard to detect ErrNotEnoughDataOnStream on client side
• pkg/stream: remove bufio and add ventryreceiver unit test
• pkg/stream: add some comments to mesasge receiver
• pkg/stream: add more corner cases guards
• pkg/stream/streamtest: add dummy file generator
• tools: fix copyright
• tools/stream: get stream content directly from immudb
• tools/stream/benchmark: add stream benchmark command
• tools/stream/benchmark/streamb: add SinceTx value to getStream

Code Refactoring

• stream kvreceiver expose Next method to iterate over key values
• stream receiver implements reader interface
• use of explicit messages for stream request
• pkg/stream: use ParseValue func in zreceiver and remove the redundant readSmallMsg func
• pkg/stream: refactor receiver to increase simplicity

Features

• add flusher to stream data to client
• add Stream Scan and client stream ServiceFactory
• Add StreamVerifiedSet and StreamVerifiedGet
• add client->server stream handler
• increase default store max value length to 32MB
• refactors and implement server->client stream handler
• Remove unnecessary dependencies (#665)
• add support for user, password and database flags in immuclient (#659)
• chunk size is passed as argument in client and server
• embedded/store: expose insertion delay while compacting
• embedded/store: configurable compaction threshold to set the min number of snapshots for a compaction to be done
• embedded/store: integrate watchers to support indexing synchronicity
• embedded/store: tx header cache to speed up indexing
• embedded/tbtree: configurable insertion delay while compaction is in progress
• embedded/tbtree: automatically set seekKey based on prefixKey when it's not set
• embedded/watchers: lightweight watching center
• embedded/watchers: fetchable current state
• pkg/client: handle illegal state error
• pkg/database: non-blocking, no history compaction
• pkg/database: non-blocking index compaction
• pkg/database: default scan parameters using up-to-date snapshot
• pkg/server: add signature on stream verifiable methods and tests
• pkg/stream: add exec all stream

Downloads

Docker image
https://hub.docker.com/r/codenotary/immudb

Immudb Binaries

File	SHA256
immudb-v0.9.2-darwin-amd64	2cd830d7f4d042121348f60b8595c3c9052a2e8d417c1a2dd4001bee40e22490
immudb-v0.9.2-linux-amd64	7481ae5dfaede6d99849a88e0202bac4ae559db6868ea7499606d9328b110307
immudb-v0.9.2-linux-amd64-static	dcbf510939a27ccf0322b0e79348f5761f7cba265f0c5e299f616db78e0aa11e
immudb-v0.9.2-linux-s390x	0dcde091392400f5aabd4fae9ff80dd19112d3cd593fa28504e2f2cb84749001
immudb-v0.9.2-windows-amd64.exe	969186c033ebb1cfbc95b9cfc0bc24d1d8d318f42797bf8db204101971544e11
immudb-v0.9.2-freebsd-amd64	c342ebb3784f6845b2250b720390f9f4d25add7927ec55a155751e217326b4c3
immudb-v0.9.2-linux-arm64	528967d2edcb55d3dfdb8d48e852fbe574e815f6e1911b2ad84250b6bb827906

Immuclient Binaries

File	SHA256
immuclient-v0.9.2-darwin-amd64	b1651986b061b7266f9fcc7e1a51680b898764e13a2d1f3bda5416c9e00df144
immuclient-v0.9.2-linux-amd64	979a2348c0e138b9706529bba9acc407551d2dc3e7efa1d484020048ececafcd
immuclient-v0.9.2-linux-amd64-static	7c9320c103a7976197eb643fa081df3500db37290d878af4d5069ece760cd2eb
immuclient-v0.9.2-linux-s390x	dfeb5edb149d2d1c0fb117c1d90e2a228ffffa12bd41327771b7974ca12f7597
immuclient-v0.9.2-windows-amd64.exe	7fdd25817f122f779a874f4a2e07a614c17d3a24b7bb79c7ad1612eaccb04245
immuclient-v0.9.2-freebsd-amd64	af4c4dc7f17b3cd312c96a4c4f42f617dd692c503fdcf7d6584aaf9558e1723f
immuclient-v0.9.2-linux-arm64	5fae3c2abb89e88784f10638b7060edf45c53c07a42768e061fc944098cbe223

File	SHA256
immuadmin-v0.9.2-darwin-amd64	93c168c837d1e135062d5d4a3463cf662eeef0ce7c8dc887740eb1f94ce9be73
immuadmin-v0.9.2-linux-amd64	70c420a0dd095f8697c06e263916d084f1da101ccc25756b241c274a0a792afd
immuadmin-v0.9.2-linux-amd64-static	b2e6d1e66ea8fd31085b0b0a32a8e26bcfcc663da050e7d372248d5144110427
immuadmin-v0.9.2-linux-s390x	58bd0edb782dc78b006538efa24ceb34c7b2b9bbdf9c84eedeac5a79a77a9c2d
immuadmin-v0.9.2-windows-amd64.exe	3103e1fdfd6d33b73f880ba13ed5807aa3df64ccf1ea211f327d9e4e84eb64af
immuadmin-v0.9.2-freebsd-amd64	af4c4dc7f17b3cd312c96a4c4f42f617dd692c503fdcf7d6584aaf9558e1723f
immuadmin-v0.9.2-linux-arm64	6969353d84e658bd11549a0f3e4945c42fd190ba9199d5a13892354f7d3c3736