Set up pixi for development by li-em · Pull Request #1674 · earth-mover/icechunk

li-em · 2026-02-18T20:46:58Z

Extends #1661 to add dependencies for running the Rust parts too.

Trying out with the code quality github actions check

Adds Pixi configuration for doc building and serving in addition to running pytest. I’ve tried tweaking the ReadTheDocs config to play nice as well. Works on #1648

li-em · 2026-02-18T21:44:52Z

.github/workflows/code-quality.yaml

      - name: Cache Dependencies
        uses: Swatinem/rust-cache@v2
        with:
          key: ${{ env.RUST_CHANNEL }}


Must fix this, rust channel doesn't make much sense anymore

@claude adapt this to the proposed changes within pixi

….8.0, rearrange and rename pixi features/envs

jbusecke · 2026-02-19T16:15:46Z

I have rearranged the logic here a bit and have now defined all developer 'tasks' in the Justfile. This seems more consistent at this point. I also updated the Python (+docs) instructions, but have not yet gotten to the Rust (and the special python test cases like upstream).

jbusecke · 2026-02-19T16:39:59Z

icechunk-python/docs/docs/ingestion/index.md

@@ -0,0 +1 @@
+# Index of ingestion


WHY IS THIS DAMN PAGE SHOWING UP ALL THE TIME?

this is my white whale

jbusecke · 2026-02-19T16:41:34Z

icechunk-python/docs/docs/contributing.md

+    ```
+    and then in the shell
+    ```bash
+    pixi run -e all init


Thinking of making this a just command too and then have a subset of the commands depend on it? Thoughts @li-em ?

Pixi shouldn't need the -e all for defined tasks as pixi run init should know which environment it should run in, and it supports dependencies for it's built in tasks via https://pixi.prefix.dev/latest/workspace/advanced_tasks/#depends-on

yup! And I think we can name the all environment as default, which also avoids the -e all argument in the CLI

Testing that now

I use a variant of that technique for OceanHackWeek, so that we can split up the dependencies by tutorial/project to track where they came from but still install them all: https://github.com/oceanhackweek/jupyter-image/blob/293f282b7a7b1b5d7bfe2f52f4c995a8fd04fdb2/py-base/pixi.toml#L155-L165

the default env works, but pixi run init does not:
I am getting

(icechunk) juliusbusecke@arm64-apple-darwin20 icechunk % pixi run init WARN Using local manifest /Users/juliusbusecke/Code/icechunk/pixi.toml rather than /Users/juliusbusecke/Code/icechunk/icechunk-python/pyproject.toml from environment variable `PIXI_PROJECT_MANIFEST` init: command not found

This works now.

pixi shell -m icechunk-python/pyproject.toml init # and then in the shell pixi run -m icechunk-python/pyproject.toml init

which still feels awkward. Is there a way I can run the task automatically when starting the shell? Or alternatively set env variables? That way I can replace the -m ... I think.

From that warning, I think you may have conflicting manifests confusing things.

Do you need to use pixi shell and maturin develop? I think re-enabling

[tool.pixi.pypi-dependencies] icechunk = { path = ".", editable = true }

With make maturin automatically run for any pixi tasks (or starting a shell).

If that's slowing some things down, icechunk could be set as a feature and then only included in the environments for specific tasks.

That was it! Thanks a lot for this!

jbusecke · 2026-02-19T16:42:51Z

icechunk-python/docs/docs/contributing.md

+    ```
+
+=== "uv"
+    TBW


Its pretty rough to keep all of these variants around. How do people feel about just instructing people to use pixi, or else they kind of have to know what they are doing? Maybe that is too exclusionary...

jbusecke · 2026-02-19T16:46:14Z

icechunk-python/pyproject.toml

    { include-group = "test" },
    "mypy",
-    "ruff",
+    "ruff==0.8.0",


I will unpin this in a separate PR, so we can cleanly separate changes in code due to the version change (see also #1647)

Justfile

pixi.lock

dcherian · 2026-02-19T17:47:58Z

I added a new stubtest check in #1675. Perhaps that should be pixi-fied too

jbusecke · 2026-02-20T16:55:57Z

Ill be OOO today and Monday, but pick this up Tues.

…into li-em/pixi

icechunk-python/docs/docs/contributing.md

jbusecke · 2026-02-26T16:18:06Z

icechunk-python/docs/docs/contributing.md

+- Use `just docs-serve --dirty` to only rebuild changed files (faster for iterative development)
 - You may need to restart if you make changes to `mkdocs.yml`
- For debugging the doc build logs, check out [docs-output-filter](https://github.com/ianhi/docs-output-filter) (you can run `uv run docs-output-filter -- mkdocs serve --livereload` once installed). *This also works to debug remote builds like RTD with the `--url` flag* 😍
+- For debugging the doc build logs, check out [docs-output-filter](https://github.com/ianhi/docs-output-filter) (you can run `docs-output-filter -- just docs-serve` once installed). *`docs-output-filter` also works to debug remote builds like RTD with the `--url` flag*


@ianhi is this ok?

icechunk-python/pyproject.toml

jbusecke · 2026-02-26T16:19:19Z

Justfile

+[doc("Run all Rust tests with RUST_LOG enabled (e.g. `just test-logs debug`)")]
 test-logs level *args='':
-  RUST_LOG=icechunk={{level}} cargo nextest run --no-fail-fast --cargo-profile {{profile}} --workspace --all-targets {{args}} -- --nocapture
+  export DYLD_LIBRARY_PATH="${CONDA_PREFIX:-}/lib" && RUST_LOG=icechunk={{level}} cargo nextest run --no-fail-fast --cargo-profile {{profile}} --workspace --all-targets {{args}} -- --nocapture


Not 100% sure if this here or above fixed the issue. Ill test locally.

Ill leave this in for now unless someone has a neater solution that does not require users to modify their mac?

jbusecke · 2026-02-26T19:22:33Z

I am happy with this for now. Ill wait for the last look of @li-em before merging. I am planning to push a couple smaller PRs soon that will:

update the ruff versions
move other parts of the CI to run through pixi/just (I can hold off on that until after IC2)
Work through the contributor guide beyond python.

abkfenris and others added 6 commits February 18, 2026 16:15

Pixi for docs and tests

1a5e954

Adds Pixi configuration for doc building and serving in addition to running pytest. I’ve tried tweaking the ReadTheDocs config to play nice as well. Works on #1648

RTD fix

d4e76b9

start adding rust deps with pixi

dcf8823

add deps needed for rust tests, and uv for the develop task

8bd8ffa

add mypy and pre-commit to dev env

00a4a35

use pixi in code-quality GHA

d15ac3f

li-em force-pushed the li-em/pixi branch from 796f51a to d15ac3f Compare February 18, 2026 20:48

fix rtd with new develop task

b2eb4a4

li-em force-pushed the li-em/pixi branch from b690763 to b2eb4a4 Compare February 18, 2026 21:30

li-em commented Feb 18, 2026

View reviewed changes

dcherian approved these changes Feb 18, 2026

View reviewed changes

jbusecke added 2 commits February 19, 2026 11:12

Move docs build to just, replace netcdf4 with h5netcdf, pin ruff to 0…

3eaf430

….8.0, rearrange and rename pixi features/envs

Update docs

1d9995c

jbusecke added 3 commits February 19, 2026 11:18

Update gha and rtd

7af04e1

remove install and use single pixi run command for the build

3c2b42b

add maturin init to rtd

15a456d

jbusecke reviewed Feb 19, 2026

View reviewed changes

jbusecke mentioned this pull request Feb 19, 2026

Bump pre-commit ruff version + define options in pyproject.toml #1647

Open

Add pixi shell alt + example how to pass args to ruff

0d78dc3

jbusecke reviewed Feb 19, 2026

View reviewed changes

Justfile Outdated Show resolved Hide resolved

Merge branch 'main' into li-em/pixi

df49b86

jbusecke mentioned this pull request Feb 19, 2026

Pixi for docs and tests #1661

Closed

abkfenris reviewed Feb 19, 2026

View reviewed changes

pixi.lock Outdated Show resolved Hide resolved

remove pixi.lock from root

70f5aa8

Switch pixi environment -> default

5e0f773

jbusecke and others added 13 commits February 24, 2026 12:15

Merge branch 'main' into li-em/pixi

729ee6e

Merge branch 'li-em/pixi' of https://github.com/earth-mover/icechunk …

97d11c6

…into li-em/pixi

update pixi.lock

88aa9cc

Remove pixi.lock binary

94b614e

use prek instead of pre-commit, add cargo-nextest

8c1bc04

Merge branch 'main' into li-em/pixi

9f7701c

Merge branch 'main' into li-em/pixi

308c0a5

Merge branch 'main' into li-em/pixi

eb281d4

Merge branch 'main' into li-em/pixi

f10e926

minor tweaks

1e037df

Some macos special sauce

28623f8

Fix pre-commit

b065baf

Added docs and just --list explainer + locally passing all python tests

1190888

jbusecke reviewed Feb 26, 2026

View reviewed changes

icechunk-python/docs/docs/contributing.md Show resolved Hide resolved

jbusecke reviewed Feb 26, 2026

View reviewed changes

icechunk-python/pyproject.toml Outdated Show resolved Hide resolved

jbusecke reviewed Feb 26, 2026

View reviewed changes

jbusecke added 6 commits February 26, 2026 11:34

Fix callout

fb85f5b

Merge branch 'main' into li-em/pixi

6beeb52

Merge branch 'main' into li-em/pixi

8f6b3ca

remove DYLD_LIBRARY_PATH shenanigans from the pixi config

db1209b

fix code quality

5b1b8e2

Update lockfile

8f20cbc

jbusecke marked this pull request as ready for review February 26, 2026 19:16

li-em merged commit 1526e16 into main Feb 26, 2026
20 checks passed

li-em deleted the li-em/pixi branch February 26, 2026 19:31

li-em mentioned this pull request Feb 26, 2026

Planning Issue: Replacing pre-commit with pixi #1648

Open

jbusecke mentioned this pull request Feb 26, 2026

Bump ruff to newest version and resulting changes #1727

Merged

Conversation

li-em commented Feb 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbusecke commented Feb 19, 2026 • edited by li-em Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dcherian commented Feb 19, 2026

Uh oh!

jbusecke commented Feb 20, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbusecke commented Feb 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jbusecke commented Feb 19, 2026 •

edited by li-em

Loading