Skip to content

Commit aef029a

Browse files
adulbrichadulbrich
committed
build: add a CMake build, and with it a WebAssembly target for dcraw_emu
LibRaw ships no CMake build of its own. README.cmake redirects to LibRaw/LibRaw-cmake, a separate repository upstream stopped supporting in October 2014, which has to be copied into the tree before it works. This adds one instead of vendoring that one. Roughly 700 of LibRaw-cmake's 737 lines are find_package() probes for LCMS, JasPer, libjpeg, zlib, OpenMP and RawSpeed, every one of which this build turns off -- so vendoring would mean carrying precisely the code most likely to misbehave under emcmake, for no benefit. There is deliberately no find_package() here at all. No source file is touched. LibRaw's optional sources self-guard behind USE_DNGSDK / USE_RAWSPEED / USE_X3FTOOLS, so never defining them is enough, and an emcmake configure succeeds first try on the unmodified tree. OpenMP staying off is load-bearing rather than incidental: threads in a wasm build mean SharedArrayBuffer, which means whoever serves the .wasm must send COOP/COEP headers. Single-threaded is what lets it sit on any static host. Validated on the 10-frame reference CR2 bracket with the exact argument list HDRICalibrationTool's pipeline constructs. Against a native build with -ffp-contract=off, output is byte-identical for bilinear (-q 0) and VNG (-q 1), and differs by 1 to 28 bytes of 67 MB for AHD (-q 3), which is the only path that calls pow() -- ahd_demosaic.cpp:46 builds a 65,536-entry cube-root LUT for CIELAB, and macOS libm and musl agree on pow to within an ulp rather than exactly. Merged through hdrgen the residual leaves mean luminance agreeing to nine significant figures. Refs radiantlab/LumiLab#237
1 parent e419de0 commit aef029a

4 files changed

Lines changed: 391 additions & 0 deletions

File tree

‎.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# CMake build trees. The names come from the recipes in FORK.md.
2+
build-native/
3+
build-native-fpc/
4+
build-wasm/
5+
build-web/
6+
7+
# Scratch space for the native-vs-wasm comparison; holds the reference CR2
8+
# bracket and ~1.3 GB of TIFF intermediates.
9+
ab/

‎CMakeLists.txt‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
cmake_minimum_required(VERSION 3.16)
2+
3+
# LibRaw ships no CMake build of its own. `README.cmake` points at
4+
# LibRaw/LibRaw-cmake, a separate repository that upstream stopped supporting in
5+
# October 2014 and that has to be copied into this tree before it works.
6+
#
7+
# This file replaces it rather than vendoring it. That is not a judgement on
8+
# LibRaw-cmake: it is that roughly 700 of its 737 lines are find_package()
9+
# probes for LCMS, JasPer, libjpeg, zlib, OpenMP and RawSpeed -- every one of
10+
# which this build turns off. Carrying that code would mean carrying precisely
11+
# the part most likely to misbehave under `emcmake`, for no benefit. What
12+
# remains once the probes are gone is a source glob and one executable.
13+
#
14+
# There is deliberately no find_package() anywhere below. Nothing outside this
15+
# repository is linked.
16+
17+
project(libraw_tools LANGUAGES CXX)
18+
19+
set(CMAKE_CXX_STANDARD 11)
20+
set(CMAKE_CXX_STANDARD_REQUIRED ON)
21+
22+
if(NOT CMAKE_BUILD_TYPE)
23+
set(CMAKE_BUILD_TYPE Release CACHE STRING "Build type")
24+
endif()
25+
26+
# --- the library --------------------------------------------------------------
27+
#
28+
# Every optional dependency is off, and stays off, by never defining its macro.
29+
# LibRaw's optional sources self-guard -- `src/integration/dngsdk_glue.cpp` and
30+
# `rawspeed_glue.cpp` are `#ifdef USE_DNGSDK` / `USE_RAWSPEED`, and `src/x3f/*`
31+
# is `#ifdef USE_X3FTOOLS` -- so they compile to nothing and need no exclusion
32+
# from the glob.
33+
#
34+
# A Canon CR2 needs none of them. Its lossless-JPEG payload is decoded by
35+
# LibRaw's own `src/decompressors/losslessjpeg.cpp` rather than by libjpeg, the
36+
# AHD demosaic that `dcraw_emu -q 3` selects lives in `src/demosaic/`, and TIFF
37+
# output is written by `src/write/` rather than by libtiff.
38+
#
39+
# Three of these stay off for licensing rather than technical reasons -- see
40+
# FORK.md.
41+
#
42+
# OpenMP is the one whose absence is load-bearing rather than incidental.
43+
# LibRaw-cmake enables it by default. Threads in a wasm build mean
44+
# SharedArrayBuffer, which means whoever serves the .wasm must send COOP/COEP
45+
# headers. Keeping this build single-threaded is what lets it be hosted on any
46+
# static host with no header configuration, which is the whole point of the
47+
# port.
48+
49+
file(GLOB_RECURSE LIBRAW_SOURCES CONFIGURE_DEPENDS
50+
"${CMAKE_CURRENT_SOURCE_DIR}/src/*.cpp")
51+
52+
# The one glob exclusion. `*_ph.cpp` are placeholder translation units whose
53+
# every function returns LIBRAW_NOT_IMPLEMENTED; they exist so LibRaw can be
54+
# built without postprocessing at all, and they define the same symbols as the
55+
# real implementations. Left in, the link fails on ten duplicate symbols --
56+
# which is the good outcome, since a resolution that happened to favour the
57+
# stubs would give a dcraw_emu whose `dcraw_process()` silently does nothing.
58+
list(FILTER LIBRAW_SOURCES EXCLUDE REGEX "_ph\\.cpp$")
59+
60+
add_library(raw STATIC ${LIBRAW_SOURCES})
61+
62+
# Sources include both "libraw/libraw.h" and "../../internal/...", so the one
63+
# include root they all resolve against is the repository root.
64+
target_include_directories(raw PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
65+
66+
# --- the tools ----------------------------------------------------------------
67+
#
68+
# One entry per sample program wanted. Adding another is one line here; the
69+
# source is already in samples/.
70+
71+
set(LIBRAW_TOOLS dcraw_emu)
72+
73+
foreach(_tool ${LIBRAW_TOOLS})
74+
add_executable(${_tool} samples/${_tool}.cpp)
75+
target_link_libraries(${_tool} PRIVATE raw)
76+
install(TARGETS ${_tool} RUNTIME DESTINATION bin)
77+
endforeach()
78+
79+
# --- WebAssembly --------------------------------------------------------------
80+
81+
if(EMSCRIPTEN)
82+
option(LIBRAW_WASM_NODERAWFS
83+
"Use the real filesystem via NODERAWFS (node only, keeps argv/paths native)" ON)
84+
85+
# LibRaw reports every I/O and format error by throwing
86+
# (LIBRAW_EXCEPTION_IO_EOF and friends) and catches in ~78 places. emcc
87+
# disables exception catching by default, which turns each of those catches
88+
# into an abort -- so a truncated or unrecognised file would take the whole
89+
# module down instead of returning an error code. Needed at compile *and*
90+
# link time.
91+
add_compile_options(-fexceptions)
92+
add_link_options(-fexceptions)
93+
94+
foreach(_tool ${LIBRAW_TOOLS})
95+
target_link_options(${_tool} PRIVATE
96+
# dcraw's unpack paths carry sizeable automatic arrays; the 64 KB
97+
# default stack is the usual cause of an unexplained early abort.
98+
-sSTACK_SIZE=5MB
99+
-sALLOW_MEMORY_GROWTH=1
100+
# wasm32 hard ceiling.
101+
-sMAXIMUM_MEMORY=4GB
102+
# Propagate the exit code. dcraw_emu returns nonzero on an
103+
# unreadable file, and the pipeline checks it.
104+
-sEXIT_RUNTIME=1
105+
)
106+
107+
if(LIBRAW_WASM_NODERAWFS)
108+
# Same argv and the same real paths as the native binary, which is
109+
# what makes native-vs-wasm a direct A/B. INVOKE_RUN is fine here:
110+
# argv already names real files, so there is nothing to stage first.
111+
target_link_options(${_tool} PRIVATE
112+
-sNODERAWFS=1 -sENVIRONMENT=node -sINVOKE_RUN=1)
113+
else()
114+
# Browser build. INVOKE_RUN must be 0 -- with it on, main() runs at
115+
# instantiation, before JS can write the RAW file into the virtual
116+
# filesystem. FS and callMain must be exported or FORCE_FILESYSTEM
117+
# compiles in a filesystem that JS cannot reach. HEAPU8 is exported
118+
# so callers can read the linear-memory high-water mark, which with
119+
# ALLOW_MEMORY_GROWTH is the only practical way to measure a run
120+
# against the wasm32 ceiling.
121+
target_link_options(${_tool} PRIVATE
122+
-sFORCE_FILESYSTEM=1
123+
-sMODULARIZE=1 -sEXPORT_ES6=1
124+
-sENVIRONMENT=web,worker
125+
-sINVOKE_RUN=0
126+
"-sEXPORTED_RUNTIME_METHODS=['FS','callMain','HEAPU8']")
127+
endif()
128+
129+
set_target_properties(${_tool} PROPERTIES SUFFIX ".js")
130+
endforeach()
131+
endif()

‎FORK.md‎

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
# radiantlab/LibRaw
2+
3+
A fork of [LibRaw/LibRaw](https://github.com/LibRaw/LibRaw) that adds a
4+
**WebAssembly** build of `dcraw_emu`, the RAW converter.
5+
6+
It exists to serve
7+
[radiantlab/HDRICalibrationTool#237](https://github.com/radiantlab/HDRICalibrationTool/issues/237),
8+
which ports that app's image pipeline off locally-installed binaries so it can
9+
run fully client-side in a browser. RAW is the workflow the app was built for:
10+
without this, a browser build would handle JPEG and TIFF only.
11+
12+
## Why a fork
13+
14+
Not because LibRaw's sources need patching. **They do not.** Every source file
15+
here is upstream's, unmodified, and an `emcmake` configure succeeds on the first
16+
attempt. LibRaw is unusually clean in this respect: its optional dependencies
17+
self-guard behind preprocessor macros, so simply not defining them is enough.
18+
19+
The fork exists because **LibRaw ships no CMake build at all**. `README.cmake`
20+
redirects to [LibRaw/LibRaw-cmake](https://github.com/LibRaw/LibRaw-cmake), a
21+
separate repository upstream stopped supporting in October 2014, which has to be
22+
copied into the tree before it works. So the build file has to live somewhere,
23+
and a fork is where it lives.
24+
25+
## The change
26+
27+
One file: **`CMakeLists.txt`**, about 120 lines. Nothing else in the tree is
28+
touched.
29+
30+
It replaces LibRaw-cmake rather than vendoring it. Roughly 700 of
31+
LibRaw-cmake's 737 lines are `find_package()` probes for LCMS, JasPer, libjpeg,
32+
zlib, OpenMP and RawSpeed, every one of which this build turns off. Vendoring
33+
would mean carrying precisely the code most likely to misbehave under
34+
`emcmake`, for nothing. What remains once the probes are gone is a source glob
35+
and one executable.
36+
37+
`CMakeLists.txt` is offered under the same terms as LibRaw itself (LGPL-2.1 or
38+
CDDL-1.0, at your option), so the fork stays coherent for anyone else who wants
39+
it.
40+
41+
### The one glob exclusion
42+
43+
`src/**/*_ph.cpp` are placeholder translation units whose every function returns
44+
`LIBRAW_NOT_IMPLEMENTED`; they exist so LibRaw can be built with no
45+
postprocessing at all, and they define the same symbols as the real
46+
implementations. Left in, the link fails on ten duplicate symbols. That is the
47+
good outcome: a link that happened to resolve in favour of the stubs would
48+
produce a `dcraw_emu` whose `dcraw_process()` silently does nothing.
49+
50+
### Every optional dependency is off
51+
52+
Off technically, because nothing needs them:
53+
54+
| Off | What it would add | Why a CR2 does not need it |
55+
|---|---|---|
56+
| libjpeg | lossy DNG | CR2's lossless-JPEG payload is decoded by `src/decompressors/losslessjpeg.cpp` |
57+
| zlib | deflate DNG | not used by CR2 |
58+
| JasPer | RedCine | not used by CR2 |
59+
| LCMS2 | ICC profile application | the pipeline does its colour work in Radiance |
60+
| libtiff | — | `dcraw_emu -T` writes TIFF through `src/write/`, not libtiff |
61+
62+
And **OpenMP**, whose absence is load-bearing rather than incidental.
63+
LibRaw-cmake enables it by default. Threads in a wasm build mean
64+
`SharedArrayBuffer`, which means whoever serves the `.wasm` must send COOP/COEP
65+
headers. Keeping this build single-threaded is what lets it be hosted on any
66+
static host with no header configuration, which is the point of the port. This
67+
matches the choice made in
68+
[radiantlab/hdrgen](https://github.com/radiantlab/hdrgen) for the same reason.
69+
70+
Off for **licensing** reasons as well as technical ones, and these should stay
71+
off unless deliberately revisited: **RawSpeed**, the **Adobe DNG SDK**,
72+
**GoPro/GPR**, **X3FTOOLS**, and the abandoned **demosaic packs** (GPL-2 and
73+
GPL-3, and not in this tree). See "Licensing" below.
74+
75+
## Branches
76+
77+
`main` only, matching the other radiantlab forks. Upstream's working branch is
78+
`master`; the `*-stable` release branches are not carried here.
79+
80+
```sh
81+
git remote add upstream https://github.com/LibRaw/LibRaw.git # once
82+
git fetch upstream master
83+
git merge upstream/master
84+
```
85+
86+
Nothing in `CMakeLists.txt` enumerates source files, so an upstream sync that
87+
adds or removes a `src/*.cpp` needs no reconciliation. The one thing to re-check
88+
after a sync is whether a new `*_ph.cpp` appeared, and to re-run the comparison
89+
below.
90+
91+
## Building
92+
93+
Requires Emscripten. Validated with `emscripten/emsdk:6.0.4`.
94+
95+
```sh
96+
# Native. Also the reference for the comparison below.
97+
cmake -S . -B build-native -DCMAKE_BUILD_TYPE=Release
98+
cmake --build build-native --target dcraw_emu -j8
99+
100+
# wasm, node -- real filesystem via NODERAWFS. Same argv and same real paths as
101+
# the native binary, which is what makes native-vs-wasm a direct A/B.
102+
emcmake cmake -S . -B build-wasm -DCMAKE_BUILD_TYPE=Release
103+
cmake --build build-wasm --target dcraw_emu -j8
104+
105+
# wasm, browser -- ES module, virtual filesystem, FS and callMain exported.
106+
emcmake cmake -S . -B build-web -DCMAKE_BUILD_TYPE=Release -DLIBRAW_WASM_NODERAWFS=OFF
107+
cmake --build build-web --target dcraw_emu -j8
108+
```
109+
110+
Both wasm variants produce a 786 KB `.wasm`.
111+
112+
## Status
113+
114+
`dcraw_emu` builds for all three targets and the wasm output is **exact except
115+
for one demosaic**.
116+
117+
### Validation against native, same tree
118+
119+
The 10-frame reference CR2 bracket (Canon EOS 5D Mark III, 8 mm fisheye,
120+
5796x3870) converted with the exact argument list HDRICalibrationTool's pipeline
121+
constructs:
122+
123+
```
124+
dcraw_emu -T -o 1 -W -j -q 3 -g 2 0 -t 0 -b 1.1 -Z <out.tiff> <in.CR2>
125+
```
126+
127+
Each output is a 67,293,432-byte 8-bit RGB TIFF. wasm runs at roughly half
128+
native speed (~2.0 s vs ~1.0 s per frame).
129+
130+
| Comparison | Result |
131+
|---|---|
132+
| wasm vs **stock native** (arm64, FMA on) | frames 1-5 identical; 6-10 differ by 16 to 704 bytes |
133+
| wasm vs native **`-ffp-contract=off`** | frames 1-5 identical; 6-10 differ by 1 to 28 bytes |
134+
| wasm run 1 vs wasm run 2 | **identical** |
135+
136+
So most of the gap is arm64 FMA contraction, exactly as found for the Radiance
137+
tools: arm64 fuses `a*b+c` into one instruction with a single rounding step and
138+
wasm MVP has no FMA, so it must round twice. The control confirms it -- stock
139+
native vs `-ffp-contract=off` native reproduces 676 of the 704 differing bytes
140+
on frame 10 on its own.
141+
142+
### The residual is AHD's cube-root table, and no flag fixes it
143+
144+
Unlike the Radiance tools, `-ffp-contract=off` does **not** get all the way to
145+
byte-identical. The remaining 1 to 28 bytes per frame are entirely attributable
146+
to the demosaic:
147+
148+
| `dcraw_emu -q` | Algorithm | wasm vs native `-ffp-contract=off` |
149+
|---|---|---|
150+
| `-q 0` | bilinear | **0 bytes differ** |
151+
| `-q 1` | VNG | **0 bytes differ** |
152+
| `-q 3` | AHD | 28 bytes differ (frame 10) |
153+
154+
Everything else was ruled out by the same method. Forcing linear gamma
155+
(`-g 1 1`) leaves the count unchanged at 28, and skipping the colour matrix
156+
(`-o 0`) leaves 23 -- so the RAW decode, white balance, colour conversion,
157+
gamma curve and TIFF writer are all bit-exact. Only AHD is not.
158+
159+
AHD is also the only path that calls `pow()`.
160+
`src/demosaic/ahd_demosaic.cpp:46` builds a 65,536-entry cube-root lookup table
161+
for the CIELAB conversion behind its homogeneity test:
162+
163+
```c
164+
cbrt[i] = r > 0.008856f ? pow(r, 1.f / 3.0f) : 7.787f * r + 16.f / 116.0f;
165+
```
166+
167+
macOS libm and Emscripten's musl libm agree on `pow` to within an ulp, not
168+
exactly. A handful of the 65,536 entries land on the other side of a rounding
169+
boundary, which flips AHD's choice of interpolation direction at the pixels
170+
where the two directions are near enough to tied. Roughly 14 pixels of 22.4
171+
million.
172+
173+
**This is a different kind of difference from FMA and it is worth being clear
174+
about.** FMA is a code-generation choice and a compiler flag removes it. A libm
175+
difference is not reachable by any flag; removing it would mean bundling a
176+
correctly-rounded `pow`. The wasm build is nonetheless **reproducible**, which
177+
is the property that actually matters: it is deterministic run to run, and
178+
because wasm MVP has no FMA it gives the same answer on every architecture and
179+
in every browser. See
180+
[radiantlab/HDRICalibrationTool#235](https://github.com/radiantlab/HDRICalibrationTool/issues/235)
181+
for the same argument applied to hdrgen's JPEG IDCT.
182+
183+
### It does not survive into the result
184+
185+
The ten TIFFs from each build merged with `hdrgen -m 1000 ... -a -e -f -g -F`:
186+
187+
```
188+
resolution 5796x3870
189+
mantissa bytes differ 44 of 67,291,560 (0.000065%)
190+
exponent bytes differ 0 of 22,430,520
191+
max abs difference 6.1e-05
192+
mean rel difference 9.5e-09
193+
mean luminance 0.135740 vs 0.135740 (relative 2.3e-09)
194+
```
195+
196+
No pixel changes exponent, and the mean luminance -- which is what becomes
197+
`COMPUTED_VERTICAL_ILLUMINANCE` downstream -- agrees to nine significant
198+
figures.
199+
200+
### Not yet done
201+
202+
Browser-tab execution. `build-web` links, and its embedding mechanism is the
203+
same one verified for the Radiance tools, but `dcraw_emu.wasm` has never been
204+
instantiated in a browser. Memory has not been measured against the wasm32
205+
ceiling either; a 5796x3870 CR2 at `-q 3` should be a few hundred MB, but that
206+
is an estimate rather than a measurement.
207+
208+
## Licensing
209+
210+
LibRaw is dual-licensed: **LGPL-2.1 or CDDL-1.0**, at the user's option
211+
(`COPYRIGHT`). This fork changes neither, and deliberately does not alter any
212+
licence notice.
213+
214+
HDRICalibrationTool is **GPL-3.0**, which settles which of the two applies:
215+
CDDL-1.0 is incompatible with the GPL, so the LGPL is the one that can be used.
216+
LGPL-2.1 §3 permits applying ordinary GPL terms to a given copy of the library,
217+
"version 2... or any later version", which is what brings the statically-linked
218+
`dcraw_emu.wasm` inside a GPL-3 work.
219+
220+
The conversion is recorded on the consuming side, not here, so this fork stays
221+
dual-licensed and useful to anyone. See
222+
[`licenses/DECISIONS.md`](https://github.com/radiantlab/HDRICalibrationTool/blob/main/licenses/DECISIONS.md)
223+
in HDRICalibrationTool.
224+
225+
The optional components listed above stay off partly for this reason. The
226+
demosaic packs in particular are GPL-2 and GPL-3 licensed, and enabling them
227+
would change the analysis. Core LibRaw's third-party pieces are all
228+
GPL-compatible: DCB and FBDD are BSD-3, the X3F reader is BSD, and the DNG SDK
229+
fragments are MIT.

‎README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,25 @@
1+
> **Note — this fork adds a WebAssembly build of `dcraw_emu`.**
2+
>
3+
> Upstream's sources need no patching at all: every optional dependency
4+
> self-guards behind a preprocessor macro, so not defining them is enough, and
5+
> an `emcmake` configure succeeds on the first attempt with the tree unmodified.
6+
> What upstream does not ship is a CMake build — `README.cmake` redirects to
7+
> [LibRaw/LibRaw-cmake](https://github.com/LibRaw/LibRaw-cmake), unsupported
8+
> since 2014 — so this fork adds one file, `CMakeLists.txt`, and changes nothing
9+
> else.
10+
>
11+
> It replaces LibRaw-cmake rather than vendoring it: ~700 of its 737 lines are
12+
> `find_package()` probes for LCMS, JasPer, libjpeg, zlib, OpenMP and RawSpeed,
13+
> all of which this build turns off. OpenMP in particular has to stay off, since
14+
> threads in wasm mean `SharedArrayBuffer` and therefore COOP/COEP headers from
15+
> whatever serves the `.wasm`.
16+
>
17+
> On the 10-frame reference CR2 bracket the wasm output is byte-identical to a
18+
> native build with `-ffp-contract=off` for every demosaic except AHD (`-q 3`),
19+
> where ~14 pixels of 22.4 million differ because macOS libm and musl disagree
20+
> on `pow()` in the last ulp. Full detail, including why that does not survive
21+
> into the merged HDR, is in [FORK.md](FORK.md).
22+
123
# LibRaw
224
## Library for reading and processing of RAW digicam images
325

0 commit comments

Comments
 (0)