Skip to content

Update docs #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 22, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,5 @@ Manifest.toml
fixture.tar.gz
fixture
.CondaPkg
*.cov
*.cov
/temp/
61 changes: 61 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# ChunkCodecs Dev Notes

## Repository directories

This monorepo holds a number of different Julia packages:

- `ChunkCodecCore`: defines the interface.
- `ChunkCodecTests`: defines tests for the interface.
- `test`: contains slower compatibility tests.

There are also a number of packages with glue code to implement the interface for various C libraries and formats.

## Running basic tests

First install Julia 1.12 or later from <https://julialang.org/install/>.

Then instantiate the workspace:

```sh
julia --project=. -e 'import Pkg; Pkg.update()'
```

Each package contains basic tests in its `"test"` sub directory.

For example here is how to run the tests for `LibZlib`.

Run test script:

```sh
julia --project=LibZlib/test LibZlib/test/runtests.jl
```

If you are on a machine with more than 24 GB of RAM you can also run:

```sh
julia --project=LibZlib/test --heap-size-hint=15G LibZlib/test/big-mem-tests.jl
```

This will use local versions of other packages in this repo.

To test the package as it would be when installed:
copy the package to a temporary directory before testing.

```sh
julia -e 'mkdir("temp"); cp("LibZlib", "temp/LibZlib")'
julia --project=temp/LibZlib -e 'import Pkg; Pkg.update()'
julia --project=temp/LibZlib/test temp/LibZlib/test/runtests.jl
```

## Running compatibility tests

The main `"test"` directory contains more tests.
These tests have complex dependencies and are more fragile.

For example to run
[imagecodecs](https://github.com/cgohlke/imagecodecs) compatibility tests.

```sh
julia --project=test -e 'import Pkg; Pkg.update()'
julia --project=test test/imagecodecs-compat.jl
```
38 changes: 17 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,30 +3,26 @@
[![CI](https://github.com/JuliaIO/ChunkCodecs.jl/actions/workflows/CI.yml/badge.svg)](https://github.com/JuliaIO/ChunkCodecs.jl/actions/workflows/CI.yml)
[![Aqua QA](https://raw.githubusercontent.com/JuliaTesting/Aqua.jl/master/badge.svg)](https://github.com/JuliaTesting/Aqua.jl)

## Warning: ChunkCodecs is currently a WIP and its API may drastically change at any time.
## Warning: There is not enough compatibility testing, so do not save important data using this package yet.
## Warning: ChunkCodecs is currently a WIP. Suggestions for major API changes are welcome.

A consistent Julia interface for lossless encoding and decoding of bytes in memory.

## Repository directories

This monorepo holds a number of different Julia packages:

- `ChunkCodecCore`: defines the interface.
- `ChunkCodecTests`: defines tests for the interface.

There are also a number of packages with glue code to implement the interface for various C libraries.

- `LibBlosc`
- `LibBzip2`
- `LibLz4`
- `LibSnappy`
- `LibZlib`
- `LibZstd`

Each package contains basic tests.

There is also a `test` directory for slower compatibility tests.
## Available Formats

| Name | Other Names | Package | Encoding | Decoding |
|---|---|---|---|---|
| Zstd | .zst RFC8878 | ChunkCodecLibZstd | ✅ | ✅ |
| Zlib | RFC1950 | ChunkCodecLibZlib | ✅ | ✅ |
| Snappy | | ChunkCodecLibSnappy | ✅ | ✅ |
| Shuffle | | ChunkCodecCore | ✅ | ✅ |
| Noop | | ChunkCodecCore | ✅ | ✅ |
| LZ4Numcodecs | | ChunkCodecLibLz4 | ✅ | ✅ |
| LZ4Frame | .lz4 | ChunkCodecLibLz4 | ✅ | ✅ |
| LZ4Block | | ChunkCodecLibLz4 | ✅ | ✅ |
| Gzip | .gz RFC1952 | ChunkCodecLibZlib | ✅ | ✅ |
| Deflate | RFC1951 | ChunkCodecLibZlib | ✅ | ✅ |
| BZ2 | .bz2 bzip2 | ChunkCodecLibBzip2 | ✅ | ✅ |
| Blosc | | ChunkCodecLibBlosc | ✅ | ✅ |

## Simple encoding and decoding

Expand Down