v0.13.0

• Fix Helm chart repo URLs because of Helm chart repostitory deprecation (see https://helm.sh/blog/charts-repo-deprecation/ for more information)

v0.12.0

• Fix type conversion bug in benji storage-usage command (issue #96)

v0.11.0

• Fix SQL generation for benji storage-usage command (issue #95)

v0.10.0

• Helm chart changes:

  • Change chart's requirements to use URL based repository references. This should help when deploying Benji via
    FluxCD's helm-operator for example. (#89)

  • Fix rendering error when specifying a nodeSelector, node affinities or tolerations. (#90)

  • Use API group rbac.authorization.k8s.io/v1 for RBAC related resources

• Add new transform module aes_256_gcm_ecc (#86)

• Add support for discovering RBD images provisioned by Ceph's CSI provisioner to the benji-backup-pvc script (#91)

v0.9.0

• Sparse blocks are no longer explicitly represented in the database. This greatly speeds up the initial step
  of backup creation if the backup is based on an older version, and a lot of the blocks are sparse. It also reduces
  database size and speeds up database operations overall due to the reduced number of rows in the blocks table. No
  database migration is necessary, but the changed representation will only apply to newly created versions.

• Implement benji storage-usage command. See the documentation on how the usage is calculated and its inherent
  limitations.

• For the rbd and rbdaio IO modules add a feature to pass all Ceph credentials as part of the URL of the
  benji backup command. This can also be used to pass other Ceph configuration options, and it is also possible
  to override the client_identifier set in Benji's configuration file. This allows Benji to connect to any
  Ceph cluster without setting up /etc/ceph/ceph.conf beforehand. See the documentation for the full URL syntax.

• benji enforce now skips removed or otherwise unavailable versions and continues with the removal.

• NBD server: The block cache is split up into a block cache and a copy-on-write store for changed blocks. For the
  block cache a maximum size can be specified and least frequently used blocks are evicted if the cache gets full.
  Some adjustments are needed to Benji's configuration due to this change. See etc/benji.yaml for an example.

v0.8.0

Notable changes:

This release contains significant changes related to the naming, format and structure of internal and external
data representations. They derive from the experience of using Benji in the last few months and from the challenges uncovered by the ongoing Kubernetes integration efforts. The changes have been bundled to avoid multiple metadata version changes and migrations.

Old metadata backups and exports and old object metadata can still be read by this version of Benji. Existing databases can be migrated to the new database structure with benji database-migrate. While this process has been tested with both PostgreSQL and SQLite it is strongly recommended to make a consistent backup of the database before attempting the migration. The migration process requires a significant amount of time and disk space when there are a lot of old backups in the database. The versions and blocks tables are completely recreated and the old data is moved over. Expect the disk usage to more than double during the migration.

• Database and metadata changes:

  • The version of metadata exports has changed from 1.1.0 to 2.0.0. Old exports (1.0.0 and 1.1.0) can still be imported.

  • snapshot_name in the versions table has been renamed to snapshot in the database and in metadata exports. The long version of the corresponding command line option has also been renamed from --snapshot-name to --snapshot.

  • name in the versions table has been renamed to volume in the database and in metadata exports.

  • bytes_dedup in the versions table has been renamed to bytes_deduplicated in the database and in metadata exports.

  • id in the blocks table has been renamed to idx in the database and in metadata exports.

  • The type of uid in the versions table has been changed from integer to string. This also affects any metadata exports. This removes the inconsistency where uid was represented as a string in some places and as an integer in others. uids are automatically generated for new versions, but there is also the option to set the uid of a version on backup via the new -u/--uid option.

  • Storages are now always represented by their name. This changes the key name in metadata exports from storage_id to storage and the corresponding value is now of type string and not of type integer anymore.

  • A new table storages is introduced to hold the internal mapping of storage ids to storage names. It is no longer necessary to specify a storage id in the configuration but existing storage ids are imported from the configuration into the database.

  • Labels are now exported as dictionaries instead of lists.

  • The letter Z has been appended to the date value in metadata exports to signify the UTC timezone.

  • These name changes also affect the specification of version filters on the command line and custom scripts might need simple adjustments.

  • The format of metadata exports is now more compact and has been optimized to facilitate efficient imports in a future version of Benji by ordering the entries in a specific way.

• Object metadata changes:

  • The object metadata version has been changed from 1.0.0 to 2.0.0. Benji can still read version 1.0.0 object metadata.

  • The timestamps in the created and modified fields of the object metadata have also been augmented with timezone information by appending a Z to the timestamp.

• The naming of automatically generated copy-on-write versions for writable NBD exports has changed.

• A workaround for a bug in various versions of nbd-client has been added. This bug leads to aborted NBD connections just after the negotiation phase was completed and leaves the NBD block device unusable. With the workaround implemented this is no longer the case.

• benji-k8s: benji-restore-pvc has been converted from Bash to Python and now runs in-cluster only. To execute it and other commands connect to the Benji maintenance pod.

• benji-k8s: Rook persistent volumes provisioned by the FlexVolume provisioner are now detected by benji-backup-pvc and can be backed up with Benji. (Contributed by @q3k. Thanks!)

• benji-k8s: The Prometheus label version_name has been renamed to volume.

v0.7.1

This release pins two package dependencies to older versions as newer releases of these dependencies broke Benji.
Fixes #49.
Fixes #51.

v0.7.0

Notable changes:

• Added a new I/O module rbdaio which uses the asynchronous API of librbd. Performance results in relation to rbd have been mixed but performance should be at least 10-20% higher on restore. In one case performance has been increased tenfold.

• Almost all Bash helper scripts have been rewritten in Python. The new scripts are calling Benji via the command line just like before. This is intentional to minimize the interdependence between Benji and these helpers. The scripts are examples only and not part of the API. There still is one example Bash script at scripts/ceph.sh to show how to interact with Benji via Bash. The helpers have additional dependencies which can be installed with pip install benji[helpers].

• The Prometheus metrics exported by benji-k8s have changed:

  • Backup metrics now longer include the auxiliary_data label.
  • Command metrics now longer include the arguments label. The arguments have been folded into the command label.

• benji-k8s: The included scripts have been replaced by Python scripts and are using the new helper modules. They should be calling compatible.

• benji-k8s: All calls to kubectl have been replaced with direct API requests. The official Python client for Kubernetes is used. kubectl is still included in the image.

• Helm chart: Volumes and volume mounts are now configurable via values.yaml. This is mostly for getting the Ceph credentials into the container but could also be used to mount file-based storage.

• Helm chart: The PostgreSQL chart dependency was updated from 2.7.6 to 4.2.2. This is the last chart which uses PostgreSQL 10 and requires no upgrade of the database data structures.

• benji-k8s and Helm chart: The image was simplified to only include the Kubernetes specific scripts and kubectl. Instead of running backups or other jobs via crond inside the container, the Helm chart now generates separate CronJobs inside of Kubernetes. This is in preparation of the move to custom resources and an operator.

• An experimental and as of yet unfinished REST API has been added. The environment variable BENJI_EXPERIMENTAL has to be set to 1 to enable the new rest-api subcommand. The API currently only services one request at a time, which limits its usefulness. The REST API has additional dependencies, they can be installed with pip install benji[rest-api].

v0.6.0 (Kubecon Barcelona Edition)

Notable changes:

• URL parsing of I/O resources is now conforming to standards. Especially for the RBD I/O module the two slashes directly after the colon are no longer valid and have to be removed (rbd://pool/image -> rbd:pool/image).

• Added I/O module for iSCSI. It is based on libiscsi and requires no elevated permissions. Please see the documentation as Benji requires a special version of the libiscsi Python bindings. The module is single-threaded and synchronous, so performance will be limited. Contributions are welcome!

• The algorithm used by benji enforce has seen an overhaul and should be more comprehensible as the time categories are based on natural time boundaries (start of the hour, day, week, month, and year) now.

• Added a restore helper script (images/benji-k8s/scripts/benji-restore-pvc) for Kubernetes. This script is intended to be run on a management system with access to the Kubernetes cluster and can restore a version into a new or an existing PVC/PV pair.

• The container images are now based on the Python 3.6 included in EPEL. The RBD support has been updated to Ceph Nautilus. Nautilus also added RADOS and RBD Python bindings for Python 3.6 which are now used instead of building them ourself.

v0.5.0

Notable changes:

• Added fsfreeze support to the benji-k8s Docker image. Just add the benji-backup.me/fsfreeze: yes annotation to the PVC. Kubernetes hosts are accessed via pods which are deployed by a DaemonSet, see the Helm chart for details.

• Use bulk inserts to speed up backups of images based on a previous version. This also decreases memory usage.

• Switched from in-memory block lists to an iterator based approach. This will increase performance and decrease memory usage when backing up large images.

• Fixed a wrong index on the blocks table. This should also increase performance. The database will need to be migrated with benji database-migrate.

• Laid the foundation for structured logging.

• Removed database table stats and assorted code and commands. Statistics are now kept together with the other version metadata in the versions table. This means they are also removed when the version is removed. If you want to keep historic statistics you need to export them beforehand with benji -m ls or benji metadata-export. This is a breaking change and you might need to adjust your scripts. As statistics are now included in a version's metadata the metadata version has changed to 1.1.0. Old metadata backups and exports with a metadata version of 1.0.0 can be imported by the current version. The statistics will be empty in that case. The database will need to be migrated with benji database-migrate.

• Fixed a bug in the time calculation of benji enforce which could lead to a late expiration of versions, the timing was a few hours off.

I'd like to thank @olifre and @adambmedent for their testing efforts!