Capnp build error on 32bit x86: Value too large for defined data type.

[tmcgilchrist]

I haven't had much opportunity to investigate further

#=== ERROR while compiling capnp-rpc-lwt.1.2.3 ================================#
# context     2.1.5 | linux/x86_32 | ocaml-base-compiler.4.14.1 | file:///home/opam/opam-repository
# path        ~/.opam/4.14/.opam-switch/build/capnp-rpc-lwt.1.2.3
# command     ~/.opam/4.14/bin/dune build -p capnp-rpc-lwt -j 71
# exit-code   1
# env-file    ~/.opam/log/capnp-rpc-lwt-7-e72851.env
# output-file ~/.opam/log/capnp-rpc-lwt-7-e72851.out
### output ###
# File "capnp-rpc-lwt/dune", line 8, characters 0-132:
#  8 | (rule
#  9 |  (targets rpc_schema.ml rpc_schema.mli)
# 10 |  (deps rpc_schema.capnp)
# 11 |  (action (run capnp compile -o %{bin:capnpc-ocaml} %{deps})))
# (cd _build/default/capnp-rpc-lwt && /usr/bin/capnp compile -o /home/opam/.opam/4.14/bin/capnpc-ocaml rpc_schema.capnp)
# kj/filesystem-disk-unix.c++:1697: warning: PWD environment variable seems invalid; pwd = /src; *e = kj/filesystem-disk-unix.c++:1693: failed: lstat(result.toString(true).cStr(), &pwdStat): Value too large for defined data type; result = src
# stack: f7d309ec
# *** Uncaught exception ***
# kj/filesystem-disk-unix.c++:305: failed: ::fstat(fd, &stats): Value too large for defined data type
# stack: f7d2cb7f 565daa7b 565db0cc 565e6419 565e67c8 f7d3e945 f7d3f912 f7d40217
# File "capnp-rpc-lwt/dune", line 13, characters 0-132:
# 13 | (rule
# 14 |  (targets persistent.ml persistent.mli)
# 15 |  (deps persistent.capnp)
# 16 |  (action (run capnp compile -o %{bin:capnpc-ocaml} %{deps})))
# (cd _build/default/capnp-rpc-lwt && /usr/bin/capnp compile -o /home/opam/.opam/4.14/bin/capnpc-ocaml persistent.capnp)
# kj/filesystem-disk-unix.c++:1697: warning: PWD environment variable seems invalid; pwd = /src; *e = kj/filesystem-disk-unix.c++:1693: failed: lstat(result.toString(true).cStr(), &pwdStat): Value too large for defined data type; result = src
# stack: f7d669ec
# *** Uncaught exception ***
# kj/filesystem-disk-unix.c++:305: failed: ::fstat(fd, &stats): Value too large for defined data type
# stack: f7d62b7f 565cea7b 565cf0cc 565da419 565da7c8 f7d74945 f7d75912 f7d76217

original logs at https://ocaml.ci.dev/github/ocurrent/ocaml-docs-ci/commit/0f917b96efc6af5be2c60ad95071033f35021927/variant/debian-12-4.14_x86_32_opam-2.1

[talex5]

The error looks like it's coming from the command /usr/bin/capnp compile -o /home/opam/.opam/4.14/bin/capnpc-ocaml persistent.capnp.

So it's either a bug in the C++ schema compiler (https://github.com/capnproto/capnproto) or, possibly, in the https://github.com/capnproto/capnp-ocaml OCaml plugin for it.

This repository's CI is showing the same problem. I wonder if the schema compiler just doesn't support x86_32? It's working on arm32.

[moyodiallo]

I noticed the error is coming from this function: https://github.com/capnproto/capnproto/blob/0274bf17374df912ea834687c667bed33bd318db/c%2B%2B/src/kj/filesystem-disk-unix.c%2B%2B#L1682-L1706

  static Path computeCurrentPath() {
    // If env var PWD is set and points to the current directory, use it. This captures the current
    // path according to the user's shell, which may differ from the kernel's idea in the presence
    // of symlinks.
    const char* pwd = getenv("PWD");
    if (pwd != nullptr) {
      Path result = nullptr;
      struct stat pwdStat, dotStat;
      KJ_IF_MAYBE(e, kj::runCatchingExceptions([&]() {
        KJ_ASSERT(pwd[0] == '/') { return; }
        result = Path::parse(pwd + 1);
        KJ_SYSCALL(lstat(result.toString(true).cStr(), &pwdStat), result) { return; }
        KJ_SYSCALL(lstat(".", &dotStat)) { return; }
      })) {
        // failed, give up on PWD
        KJ_LOG(WARNING, "PWD environment variable seems invalid", pwd, *e);
      } else {
        if (pwdStat.st_ino == dotStat.st_ino &&
            pwdStat.st_dev == dotStat.st_dev) {
          return kj::mv(result);
        } else {
          KJ_LOG(WARNING, "PWD environment variable doesn't match current directory", pwd);
        }
      }
    }

To try to reproduce, run:

$ ocluster-client submit-obuilder \
      https://github.com/mirage/capnp-rpc.git a21ddc6fca2ae8e6c5f59a66f7308284af3eecfb \
      --connect ~/submission.cap --pool linux-x86_64 \
      --local-file capnp.spec

with the capnp.spec file, I added some commands: stat /src , stat . for the informations about lstat function.

$ cat capnp.spec
((from ocaml/opam:debian-12-ocaml-4.14@sha256:f82565568872bd53dffaf29565b967cb872186142ae20657233f6d228761b989)
 (comment debian-12-4.14_x86_32_opam-2.1)
 (user (uid 1000) (gid 1000))
 (shell /usr/bin/linux32 /bin/sh -c)
 (env CLICOLOR_FORCE 1)
 (env OPAMCOLOR always)
 (workdir /src)
 (run (shell "ls -lah /src"))
 (run (shell "df -i"))
 (run (shell "echo $PWD"))
 (run (shell "stat /src"))
 (run (shell "stat ."))
 (run (shell "stat /"))
 (run (shell "sudo ln -f /usr/bin/opam-2.1 /usr/bin/opam"))
 (run (shell "opam init --reinit -ni"))
 (run (shell "opam exec -- ocaml -version && opam --version"))
 (workdir /src)
 (run (shell "sudo chown opam /src"))
 (run (cache (opam-archives (target /home/opam/.opam/download-cache)))
      (network host)
      (shell "cd ~/opam-repository && (git cat-file -e 3856864b3e9cf7fc260757d7ba24ca7bb0a236e3 || git fetch origin master) && git reset -q --hard 3856864b3e9cf7fc260757d7ba24ca7bb0a236e3 && git log --no-decorate -n1 --oneline && opam update -u"))
 (copy (src capnp-rpc.opam capnp-rpc-unix.opam capnp-rpc-net.opam capnp-rpc-mirage.opam capnp-rpc-lwt.opam)
       (dst ./))
 (run (network host)
      (shell  "opam pin add -yn capnp-rpc.dev './' && \
             \nopam pin add -yn capnp-rpc-unix.dev './' && \
             \nopam pin add -yn capnp-rpc-net.dev './' && \
             \nopam pin add -yn capnp-rpc-mirage.dev './' && \
             \nopam pin add -yn capnp-rpc-lwt.dev './'"))
 (env DEPS "afl-persistent.1.4 alcotest.1.7.0 alcotest-lwt.1.7.0 angstrom.0.15.0 arp.3.1.0 asetmap.0.8.1 asn1-combinators.0.2.6 astring.0.8.5 base.v0.16.3 base-bigarray.base base-bytes.base base-threads.base base-unix.base base64.3.5.1 bigarray-compat.1.1.0 bigstringaf.0.9.1 bos.0.2.1 ca-certs-nss.3.92 camlp-streams.5.0.1 capnp.3.6.0 cmdliner.1.2.0 conf-capnproto.2 conf-gmp.4 conf-gmp-powm-sec.3 cppo.1.6.9 csexp.1.5.2 cstruct.6.2.0 cstruct-lwt.6.2.0 dns.7.0.3 dns-client.7.0.3 dns-client-mirage.7.0.3 domain-name.0.4.0 dune.3.10.0 dune-configurator.3.10.0 duration.0.2.1 eqaf.0.9 ethernet.3.2.0 extunix.0.4.1 fmt.0.9.0 fpath.0.7.3 gmap.0.3.0 happy-eyeballs.0.6.0 hkdf.1.0.4 io-page.2.3.0 io-page-unix.2.3.0 ipaddr.5.5.0 ipaddr-cstruct.5.5.0 logs.0.7.0 lru.0.3.1 lwt.5.7.0 lwt-dllist.1.0.1 macaddr.5.5.0 macaddr-cstruct.5.5.0 mdx.2.3.0 metrics.0.4.1 mirage-clock.4.2.0 mirage-crypto.0.11.1 mirage-crypto-ec.0.11.1 mirage-crypto-pk.0.11.1 mirage-crypto-rng.0.11.1 mirage-crypto-rng-lwt.0.11.1 mirage-flow.3.0.0 mirage-kv.6.1.1 mirage-net.4.0.0 mirage-profile.0.9.1 mirage-random.3.0.0 mirage-time.3.0.0 mirage-vnetif.0.6.0 mtime.2.0.0 num.1.4 ocaml.4.14.1 ocaml-base-compiler.4.14.1 ocaml-compiler-libs.v0.12.4 ocaml-config.2 ocaml-options-vanilla.1 ocaml-syntax-shims.1.0.0 ocaml-version.3.6.1 ocamlbuild.0.14.2 ocamlfind.1.9.6 ocplib-endian.1.2 odoc-parser.2.0.0 optint.0.3.0 parsexp.v0.16.0 pbkdf.1.2.0 ppx_cstruct.6.2.0 ppx_derivers.1.2.1 ppxlib.0.30.0 prometheus.1.2 psq.0.2.1 ptime.1.1.0 randomconv.0.1.3 re.1.11.0 res.5.0.1 result.1.5 rresult.0.7.0 seq.base sexplib.v0.16.0 sexplib0.v0.16.0 stdint.0.7.2 stdio.v0.16.0 stdlib-shims.0.3.0 stringext.1.6.0 tcpip.8.0.0 tls.0.17.1 tls-mirage.0.17.1 topkg.1.0.7 uri.4.2.0 uutf.1.0.3 x509.0.16.5 zarith.1.13")
 (env CI true)
 (env OCAMLCI true)
 (run (cache (opam-archives (target /home/opam/.opam/download-cache)))
      (network host)
      (shell "opam update --depexts && opam install --cli=2.1 --depext-only -y capnp-rpc.dev capnp-rpc-unix.dev capnp-rpc-net.dev capnp-rpc-mirage.dev capnp-rpc-lwt.dev $DEPS"))
 (run (cache (opam-archives (target /home/opam/.opam/download-cache)))
      (network host)
      (shell "opam install $DEPS"))
 (copy (src .) (dst /src))
 (run (shell "ls -lah /src"))
 (run (shell "df -i"))
 (run (shell "stat /src"))
 (run (shell "stat ."))
 (run (shell "opam exec -- dune build @install @check @runtest && rm -rf _build"))
)

I noticed in some machines, like toxis, clete,laodoke, it is building fine but some others failing like cree,marpe,iphito.
I remark some pattern about their inode number. It seems when the inode number near or exceed for sure 2^32-1, the build fails.

From marpe machine (inode 16143284239):

....
/src: (run (shell "stat /src"))
  File: /src
  Size: 560       	Blocks: 0          IO Block: 4096   directory
Device: 0,59	Inode: 16146239863  Links: 1
Access: (0755/drwxr-xr-x)  Uid: ( 1000/    opam)   Gid: ( 1000/    opam)
Access: 2023-09-09 11:49:24.793995419 +0100
Modify: 2023-09-09 11:49:24.000000000 +0100
Change: 2023-09-09 11:53:12.594973054 +0100
 Birth: -
....
Failed: Build failed.

From asteria (inode 17423550515):

...
/src: (run (shell "stat /src"))
  File: /src
  Size: 40        	Blocks: 0          IO Block: 4096   directory
Device: 0,52	Inode: 17423550515  Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-09-09 11:56:40.301145705 +0100
Modify: 2023-09-09 11:56:40.293145678 +0100
Change: 2023-09-09 11:56:40.293145678 +0100
 Birth: -
...
Failed: Build failed.

From toxis (inode 257599):

...
src: (run (shell "stat ."))
  File: .
  Size: 534       	Blocks: 0          IO Block: 4096   directory
Device: 0,94	Inode: 257599      Links: 1
Access: (0755/drwxr-xr-x)  Uid: ( 1000/    opam)   Gid: ( 1000/    opam)
Access: 2023-09-09 12:10:58.161471501 +0100
Modify: 2023-09-09 12:11:52.038188339 +0100
Change: 2023-09-09 12:11:52.038188339 +0100
 Birth: 2023-09-09 11:07:57.162765685 +0100
...
Job succeeded

From x86-bm-c5 (inode 6735750119):

...
/src: (run (shell "stat /src"))
  File: /src
  Size: 40        	Blocks: 0          IO Block: 4096   directory
Device: 0,85	Inode: 6735750119  Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-09-09 12:19:19.379131633 +0100
Modify: 2023-09-09 12:19:19.371131495 +0100
Change: 2023-09-09 12:19:19.371131495 +0100
 Birth: -
...
Failed: Build failed.

[MisterDA]

After investigating with @moyodiallo, we've discovered that capnproto has not defined the _FILE_OFFSET_BITS=64 macro on the 0.9.2 version. It is however present in v1.0.x, so this bug is fixed. Debian still uses 0.9.2.

_FILE_OFFSET_BITS allows the 64 bit interface to replace the old interface

Types such as off_t and ino_t are 64 bits on 32 bits systems if the macro is defined. I've reported a bug on Debian and I hope they fix it. If the value (e.g., the size of the file or the inode number) exceeds 32 bits, then we get the error.

[moyodiallo]

This PR capnproto/capnproto#1824 fixes the bug on capnproto, even if the bug on Debian is fixed by an upgrade the bug will still remain.

[moyodiallo]

The PR capnproto/capnproto#1824 that I opened was closed without being merged. And I opened another one to be more specific/precise and finally merged now capnproto/capnproto#1830.