Why is arrow a peer dependency?

[domoritz]

Shouldn't it be a normal dependency with a flexible range of allowed versions? Having two package versions seems fishy.

[jraymakers]

We found that when different parts of an application depended on apache-arrow directly, multiple copies of this package could get pulled in. If this happens, breaking bugs can occur. Several parts of apache-arrow use instanceof to check the type of objects (e.g. RecordBatch, Table). If multiple copies of the library are in use, multiple instances of each type object exist, so these checks can unexpectedly return false when comparing two semantically-equivalent but distinct copies of the "same" type.

By making apache-arrow a peer dependency, with a range that matches the version of apache-arrow used by the version of duckdb-wasm we depend on, we ensure there is only a single copy around.

[domoritz]

What if instead you make it a dependency but you allow any version (*)? Would the bundler resolve the dependency to a single version rather than a copy?

I guess the underlying issue is that arrow does not follow semantic versioning.

[jraymakers]

We do depend on some fixes & features in the newer versions, so "*" isn't really correct. I'm not sure whether it would resolve the multi-copy issue. It's hard to reason about the many different dependency and bundler configurations that applications might use.

The updated SDK I'm working on changes the dependency structure. (In particular, we'll depend on duckdb-wasm using NPM, instead of using a version built & hosted by MotherDuck.) As I work on that, I'll investigate whether there's a simpler solution to this problem that doesn't require peer dependencies.