Skip to content

Latest commit

 

History

History
299 lines (233 loc) · 12.9 KB

File metadata and controls

299 lines (233 loc) · 12.9 KB

radiantlab/LibRaw

A fork of LibRaw/LibRaw that adds a WebAssembly build of dcraw_emu, the RAW converter.

It exists to serve radiantlab/HDRICalibrationTool#237, which ports that app's image pipeline off locally-installed binaries so it can run fully client-side in a browser. RAW is the workflow the app was built for: without this, a browser build would handle JPEG and TIFF only.

Why a fork

Not because LibRaw's sources need patching. They do not. Every source file here is upstream's, unmodified, and an emcmake configure succeeds on the first attempt. LibRaw is unusually clean in this respect: its optional dependencies self-guard behind preprocessor macros, so simply not defining them is enough.

The fork exists because LibRaw ships no CMake build at all. README.cmake redirects to LibRaw/LibRaw-cmake, a separate repository upstream stopped supporting in October 2014, which has to be copied into the tree before it works. So the build file has to live somewhere, and a fork is where it lives.

The change

One file: CMakeLists.txt, about 120 lines. Nothing else in the tree is touched.

It replaces LibRaw-cmake rather than vendoring it. Roughly 700 of LibRaw-cmake's 737 lines are find_package() probes for LCMS, JasPer, libjpeg, zlib, OpenMP and RawSpeed, every one of which this build turns off. Vendoring would mean carrying precisely the code most likely to misbehave under emcmake, for nothing. What remains once the probes are gone is a source glob and one executable.

CMakeLists.txt is offered under the same terms as LibRaw itself (LGPL-2.1 or CDDL-1.0, at your option), so the fork stays coherent for anyone else who wants it.

The one glob exclusion

src/**/*_ph.cpp are placeholder translation units whose every function returns LIBRAW_NOT_IMPLEMENTED; they exist so LibRaw can be built with no postprocessing at all, and they define the same symbols as the real implementations. Left in, the link fails on ten duplicate symbols. That is the good outcome: a link that happened to resolve in favour of the stubs would produce a dcraw_emu whose dcraw_process() silently does nothing.

Every optional dependency is off

Off technically, because nothing needs them:

Off What it would add Why a CR2 does not need it
libjpeg lossy DNG CR2's lossless-JPEG payload is decoded by src/decompressors/losslessjpeg.cpp
zlib deflate DNG not used by CR2
JasPer RedCine not used by CR2
LCMS2 ICC profile application the pipeline does its colour work in Radiance
libtiff dcraw_emu -T writes TIFF through src/write/, not libtiff

And OpenMP, whose absence is load-bearing rather than incidental. LibRaw-cmake enables it by default. Threads in a wasm build mean SharedArrayBuffer, which means whoever serves the .wasm must send COOP/COEP headers. Keeping this build single-threaded is what lets it be hosted on any static host with no header configuration, which is the point of the port. This matches the choice made in radiantlab/hdrgen for the same reason.

Off for licensing reasons as well as technical ones, and these should stay off unless deliberately revisited: RawSpeed, the Adobe DNG SDK, GoPro/GPR, X3FTOOLS, and the abandoned demosaic packs (GPL-2 and GPL-3, and not in this tree). See "Licensing" below.

Branches

Work happens on main, which is the default branch. Upstream's working branch is master, renamed here to match the other radiantlab forks.

Upstream's twelve *-stable release branches came across with the fork and are not used. They can go whenever convenient:

for b in 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.20 0.21 0.22; do
  git push origin --delete "$b-stable"
done
git remote add upstream https://github.com/LibRaw/LibRaw.git   # once
git fetch upstream master
git merge upstream/master

Nothing in CMakeLists.txt enumerates source files, so an upstream sync that adds or removes a src/*.cpp needs no reconciliation. The one thing to re-check after a sync is whether a new *_ph.cpp appeared, and to re-run the comparison below.

Building

Requires Emscripten. Validated with emscripten/emsdk:6.0.4.

# Native. Also the reference for the comparison below.
cmake -S . -B build-native -DCMAKE_BUILD_TYPE=Release
cmake --build build-native --target dcraw_emu -j8

# wasm, node -- real filesystem via NODERAWFS. Same argv and same real paths as
# the native binary, which is what makes native-vs-wasm a direct A/B.
emcmake cmake -S . -B build-wasm -DCMAKE_BUILD_TYPE=Release
cmake --build build-wasm --target dcraw_emu -j8

# wasm, browser -- ES module, virtual filesystem, FS and callMain exported.
emcmake cmake -S . -B build-web -DCMAKE_BUILD_TYPE=Release -DLIBRAW_WASM_NODERAWFS=OFF
cmake --build build-web --target dcraw_emu -j8

Both wasm variants produce an 837 KB .wasm.

Status

dcraw_emu builds for all three targets and the wasm output is exact except for one demosaic.

Validation against native, same tree

The 10-frame reference CR2 bracket (Canon EOS 5D Mark III, 8 mm fisheye, 5796x3870) converted with the exact argument list HDRICalibrationTool's pipeline constructs:

dcraw_emu -T -o 1 -W -j -q 3 -g 2 0 -t 0 -b 1.1 -Z <out.tiff> <in.CR2>

Each output is a 67,293,432-byte 8-bit RGB TIFF. wasm runs at roughly half native speed (~2.0 s vs ~1.0 s per frame).

Comparison Result
wasm vs stock native (arm64, FMA on) frames 1-5 identical; 6-10 differ by 16 to 704 bytes
wasm vs native -ffp-contract=off frames 1-5 identical; 6-10 differ by 1 to 28 bytes
wasm run 1 vs wasm run 2 identical
wasm under node vs wasm in a browser identical
stock native vs the dcraw_emu binary HDRICalibrationTool shipped in May 2025 identical

That last row matters for the cutover. This tree is 82 commits past LibRaw 0.22.0 and the shipped binary is over a year old, so the two could easily have disagreed on TIFF metadata or decode. They do not, byte for byte, which means replacing the bundled binary with a build from this fork changes nothing for existing users on the native path.

So most of the gap is arm64 FMA contraction, exactly as found for the Radiance tools: arm64 fuses a*b+c into one instruction with a single rounding step and wasm MVP has no FMA, so it must round twice. The control confirms it -- stock native vs -ffp-contract=off native reproduces 676 of the 704 differing bytes on frame 10 on its own.

The residual is AHD's cube-root table, and no flag fixes it

Unlike the Radiance tools, -ffp-contract=off does not get all the way to byte-identical. The remaining 1 to 28 bytes per frame are entirely attributable to the demosaic:

dcraw_emu -q Algorithm wasm vs native -ffp-contract=off
-q 0 bilinear 0 bytes differ
-q 1 VNG 0 bytes differ
-q 3 AHD 28 bytes differ (frame 10)

Everything else was ruled out by the same method. Forcing linear gamma (-g 1 1) leaves the count unchanged at 28, and skipping the colour matrix (-o 0) leaves 23 -- so the RAW decode, white balance, colour conversion, gamma curve and TIFF writer are all bit-exact. Only AHD is not.

AHD is also the only path that calls pow(). src/demosaic/ahd_demosaic.cpp:46 builds a 65,536-entry cube-root lookup table for the CIELAB conversion behind its homogeneity test:

cbrt[i] = r > 0.008856f ? pow(r, 1.f / 3.0f) : 7.787f * r + 16.f / 116.0f;

macOS libm and Emscripten's musl libm agree on pow to within an ulp, not exactly. A handful of the 65,536 entries land on the other side of a rounding boundary, which flips AHD's choice of interpolation direction at the pixels where the two directions are near enough to tied. Roughly 14 pixels of 22.4 million.

This is a different kind of difference from FMA and it is worth being clear about. FMA is a code-generation choice and a compiler flag removes it. A libm difference is not reachable by any flag; removing it would mean bundling a correctly-rounded pow. The wasm build is nonetheless reproducible, which is the property that actually matters: it is deterministic run to run, identical under node and in a browser, and the same on every architecture. The build passes -ffp-contract=off so that last part is stated rather than inherited -- wasm MVP has no FMA, but relaxed SIMD's f32x4.relaxed_madd may fuse at the engine's discretion, so the guarantee should not rest on emcc happening not to vectorise. See radiantlab/HDRICalibrationTool#235 for the same argument applied to hdrgen's JPEG IDCT.

It does not survive into the result

The ten TIFFs from each build merged with hdrgen -m 1000 ... -a -e -f -g -F:

resolution            5796x3870
mantissa bytes differ 44 of 67,291,560  (0.000065%)
exponent bytes differ 0 of 22,430,520
max abs difference    6.1e-05
mean rel difference   9.5e-09
mean luminance        0.135740  vs  0.135740   (relative 2.3e-09)

No pixel changes exponent, and the mean luminance -- which is what becomes COMPUTED_VERTICAL_ILLUMINANCE downstream -- agrees to nine significant figures.

In a browser, on a real CR2

build-web was driven in headless Chromium against capt10.CR2, staged into MEMFS by JS and run through callMain with the pipeline's argument list:

fetched CR2:               26,010,200 bytes
heap after instantiate:    16.0 MiB
heap after staging input:  16.0 MiB
exit code:                 0   (1.9 s)
heap high-water mark:      266.1 MiB   (6.5% of the wasm32 ceiling)
output TIFF:               67,293,432 bytes
sha256:                    8137c98ac37c930c3fdaf5be0fc1d17fe40985dc5faa7411dc06d1f6a78af2ab

Three things worth keeping:

  • The browser output is byte-identical to the NODERAWFS build, same digest. The virtual filesystem changes nothing about the numbers.
  • 266 MiB peak, a twelfth of the wasm32 budget. Memory was the open risk in #237; it is not a constraint. Frames are converted one at a time, and each stage gets a fresh module instance, so this is the whole per-frame cost.
  • Staging the 26 MB input did not grow the heap at all, 16.0 MiB before and after. MEMFS keeps file bytes outside wasm linear memory, confirming what #234 found for the Radiance tools. Input size does not compete with the working set.

Ten instances do not accumulate

266 MiB is per instance, and a consumer creates a fresh one per frame because EXIT_RUNTIME=1 allows only one main() per module. Since a heap never shrinks within an instance, the real question is whether ten frames means ten live heaps. Ten sequential conversions in one tab:

run  1  exit 0  2.0s  wasm heap 266.1 MiB  js heap 284.8 MiB  digest same
run  2  exit 0  1.5s  wasm heap 266.1 MiB  js heap 348.3 MiB  digest same
run  3  exit 0  1.5s  wasm heap 266.1 MiB  js heap 655.6 MiB  digest same
run  4  exit 0  1.5s  wasm heap 266.1 MiB  js heap 442.8 MiB  digest same
...
run 10  exit 0  1.5s  wasm heap 266.1 MiB  js heap 592.0 MiB  digest same

They do not. The wasm heap is flat at 266.1 MiB, because each instance gets its own, and the JS heap oscillates between 285 and 700 MiB rather than climbing (655 down to 442, 699 down to 516). Non-monotonic is the signature of instances being collected between frames. A ten-frame bracket peaks near 700 MiB, not the 2.6 GB ten retained heaps would cost.

Every run produced the same digest, so nothing about instance reuse perturbs the numbers either.

Not yet done

Nothing in this repository. Wiring dcraw_emu into HDRICalibrationTool's TypeScript pipeline so RAW inputs actually reach it is tracked there.

Licensing

LibRaw is dual-licensed: LGPL-2.1 or CDDL-1.0, at the user's option (COPYRIGHT). This fork changes neither, and deliberately does not alter any licence notice.

HDRICalibrationTool is GPL-3.0, which settles which of the two applies: CDDL-1.0 is incompatible with the GPL, so the LGPL is the one that can be used. LGPL-2.1 §3 permits applying ordinary GPL terms to a given copy of the library, "version 2... or any later version", which is what brings the statically-linked dcraw_emu.wasm inside a GPL-3 work.

The conversion is recorded on the consuming side, not here, so this fork stays dual-licensed and useful to anyone. See licenses/DECISIONS.md in HDRICalibrationTool.

The optional components listed above stay off partly for this reason. The demosaic packs in particular are GPL-2 and GPL-3 licensed, and enabling them would change the analysis. Core LibRaw's third-party pieces are all GPL-compatible: DCB and FBDD are BSD-3, the X3F reader is BSD, and the DNG SDK fragments are MIT.