v5.8.2+man1

In this release we have dropped support for Python 3.7 (#2400). If you are a paying customer and require Python 3.7 support, please get in touch with us at [email protected] or in our Slack.

🐛 Fixes

• 9220057136: Allow block manager consolidation in Pandas < 2 (#2392)
  • By default, Pandas will "consolidate" consecutive columns of the same dtype (e.g. int64) from the multiple buffers returned by ArcticDB, into a single larger buffer. This can have performance benefits for short, wide dataframes. However, ArcticDB is optimised for longer, narrower dataframes, and so the performance penalty (introducing an additional memcpy) has historically been bypassed in ArticDB to improve reading speed. Unfortunately, we discovered recently that certain Pandas dataframe processing operations, such as replace, have bugs when consolidation is disallowed for Pandas versions <2.0.0. As such, the consolidation bypassing behaviour is now disabled in ArcticDB when running with Pandas versions <2.0.0. The same bugs were not observed in Pandas versions >= 2.0.0, and so the behaviour in this case remains unchanged.

💻 Internal Changes

• Upgrade GCC to 11.2 and add sparrow dependency (#2355)

---

The wheels are on PyPI. Below are for debugging:

v5.8.1+man0

🐛 Fixes

• Fix link to the SL caching docs in warning (#2380)
• Fix s3 authentication test (#2381)
• Fix moto teardown rare problem (#2383)
• Change delete to load only undeleted versions (#2375)
• Docs 9289736269: Add API docs for where and concat (#2388)
• analyze failures for storage locks (#2379)

---

The wheels are on PyPI. Below are for debugging:

v5.8.0

🚀 Features

• Query Stats s3 v3 (#2309)
• Implement delete versions funcationality (#2352)

🐛 Fixes

• Bugfix 9209057536: Allow concatenation of uint64 columns with int* columns (#2365)
• Try to fix flaky compat tests (#2364)
• Fix for scheduled workflow build failures to appear in slack (#2371)
• New versions 5.5 and 5.6 added to installation tests monitoring (#2373)
• Extend test coverage for recursive normalizers (#2366)
• Respect legacy env vars (#2376)
• Query Stats s3 v3 benchmark (#2346)
• Update recursive normalizers layout assertions (#2377)

---

The wheels are on PyPI. Below are for debugging:

v5.8.0+man0

🚀 Features

• Query Stats for S3 storage (#2309). See the documentation. This lets you gather statistics about the IO operations performed by ArcticDB:

import arcticdb as adb
import arcticdb.toolbox.query_stats as qs

arctic = adb.Arctic(f"s3://{HOST}:{BUCKET}?access={ACCESS}&secret={SECRET}")
lib = arctic["library_name"]

# Collect statistics for specific operations
with qs.query_stats():
    lib.list_symbols()
    
# Get the collected statistics
stats = qs.get_query_stats()
print(stats)

🏎️ Performance

• Implement delete versions functionality (#2352)
  • This changes the versions kwarg on delete so that it can delete several versions of a symbol in a fixed number of IO operations, whereas it previously performed an IO operation for each version.

🐛 Fixes

• Allow concatenation of uint64 columns with int columns of any width (#2365)

---

The wheels are on PyPI. Below are for debugging:

v5.7.0

🚀 Features

• Introduce a ternary operator for filters and projections (#2103)

The new ternary operator is designed to work like numpy.where, allowing operations like:

q = QueryBuilder()
q = q.apply("new_col", where(q["col1"] == 0, q["col2"], q["col3"]))

For more details, see the docstring and associated tests.

🐛 Fixes

• Bug fixes for resampling (#2351)
  • Fix crash when resampling data that was updated with some particular old ArcticDB client versions
  • Fix resampling over a date_range not covered by the symbol's data
  • Better error message when resampling a non-timeseries indexed symbol
• Fix for reading multiindex series with timestamps as the first level and strings as the second level (#2363)
  • This regressed in 5.6.0

📖 Docs

• Add concat demo notebook (#2361). This is available on our docs site here.

---

The wheels are on PyPI. Below are for debugging:

v5.7.0+man0

🚀 Features

• Enhancement/7992967434/filters and projections ternary operator (#2103)

🐛 Fixes

• nfs backed tests for v1 API (#2350)
• Bugfix/9123099670/fix resampling of old updated data (#2351)
• Schedule S3 tests and fix STS to run only against AWS S3 (#2356)
• Update installation_tests.yml with 5.3 and 5.4 final versions (#2362)
• Docs 8975554293: Add concat demo notebook (#2361)
• Fix multiindex series (#2363)
• Fix gcp lib unreachable after making it read only (#2349)

---

The wheels are on PyPI. Below are for debugging:

v5.6.0

🚀 Features

• Disable s3 checksumming (#2337)
• Enhancement 8277989680: symbol concatenation (#2142)

🐛 Fixes

• Asv v2 s3 tests (Refactored) (#2249)

• Installation tests v4 (#2339)

• Installation Tests Workflow Fixes (#2354)

• Add resolve_defaults back in as a static method of NativeVersionStore (#2358)

• Patch for Real S3 library names (#2353)

• File operations (#2345)

---

The wheels are on PyPI. Below are for debugging:

v5.6.0+man1

🚀 Features

• Disable s3 checksumming (#2337)
• Enhancement 8277989680: symbol concatenation (#2142)

🐛 Fixes

• Asv v2 s3 tests (Refactored) (#2249)

• Installation tests v4 (#2339)

• Installation Tests Workflow Fixes (#2354)

• Add resolve_defaults back in as a static method of NativeVersionStore (#2358)

• Patch for Real S3 library names (#2353)

• File operations (#2345)

---

The wheels are on PyPI. Below are for debugging:

v5.6.0+man0

🚀 Features

• Disable s3 checksumming (#2337)
• Enhancement 8277989680: symbol concatenation (#2142)

🐛 Fixes

• Asv v2 s3 tests (Refactored) (#2249)

• Installation tests v4 (#2339)

• Installation Tests Workflow Fixes (#2354)

• Add resolve_defaults back in as a static method of NativeVersionStore (#2358)

• Patch for Real S3 library names (#2353)

• File operations (#2345)

---

The wheels are on PyPI. Below are for debugging:

v5.5.1

🐛 Fixes

• Fix permissions for the automatic docs building (#2347)
• Delete STS roles regularly (#2344)

---

The wheels are on PyPI. Below are for debugging: