help building for alpine

[toxic0berliner]

Hello,

I've been trying to build duc for alpine to run it as a minimal container.
I do have it running on debian:12 quite smoothely, but I'm stuck with alpine, compile looks good, but I'm gettin gthe following error when trying to index anything :

Error opening: /duc/.duc.db - Unknown error, contact the author
Unknown error, contact the author

I am using my own dockerfile available here, as you can see I have tried to get all dependencies and had to build tkrzw from source. Maybe my using the master branch of tkrzw is the issue and you have a dependency to a specific version ?

Thanks in advance for any kind of help sorting this out.

[l8gravely]

>>>> "toxic0berliner" == toxic0berliner ***@***.***> writes: Hello, I've been trying to build duc for alpine to run it as a minimal container. I do have it running on debian:12 quite smoothely, but I'm stuck with alpine, compile looks good, but I'm gettin gthe following error when trying to index anything : Error opening: /duc/.duc.db - Unknown error, contact the author Unknown error, contact the author

Can you tell me which version of duc you're trying to use? ./duc --version would help. I do have some bugs in the code right now, you're going to get a bunch of garbage when you use --version, which I need to fix.

I am using my own dockerfile available here, as you can see I have tried to get all dependencies and had to build tkrzw from source. Maybe my using the master branch of tkrzw is the issue and you have a dependency to a specific version ?

So you're using v1.5.0-rc1 then, which is good!

Thanks in advance for any kind of help sorting this out.

Some hints on how to setup and use docker for testing, I'm clueless. And what version of 'alpine' are you using? Can you run your build of duc on alpine after building but before putting it into a docker container? And since you're using tkrzw, could you use the utilities to check the file? tkrzw_dbm_util inspect --dbm hash /duc/.duc.db and send that results as well? John

[toxic0berliner]

Thanks a lot for your quick feedback !

Can you tell me which version of duc you're trying to use? [...] you're going to
get a bunch of garbage when you use --version, which I need to fix.

The version is indeed 1.5.0-rc1 but there is not much garbage in the output 😄 :

f5b996fee160:~# duc --version
Unknown option 'database'
duc version: 1.5.0-rc1
options: cairo x11 ui tkrzw

And since you're using tkrzw, could you use the utilities to check the
file?

Sure, I tried this to recreate the issue and get a db file that it claims it can't open. _{(click to unfold and see my exact commands to build this db & the error again)}

f5b996fee160:~# pwd
/duc
f5b996fee160:~# ls -alh ./
total 16K
drwxrwxrwx    1 root     root        4.0K Oct 28 21:35 .
drwxr-xr-x    1 root     root        4.0K Oct 30 10:11 ..
drwxr-xr-x    2 root     root        4.0K Oct 28 21:35 db
-rw-r--r--    1 root     root        1.8K Oct 28 21:25 duc_startup.sh
f5b996fee160:~# ls -alh /tmp
total 8K
drwxrwxrwt    1 root     root        4.0K Oct 28 21:35 .
drwxr-xr-x    1 root     root        4.0K Oct 30 10:11 ..
f5b996fee160:~# duc index /tmp -d ./tmp.db
Error opening: ./tmp.db - Unknown error, contact the author
Unknown error, contact the author
f5b996fee160:~# ls -alh
total 5M
drwxrwxrwx    1 root     root        4.0K Oct 30 10:17 .
drwxr-xr-x    1 root     root        4.0K Oct 30 10:11 ..
drwxr-xr-x    2 root     root        4.0K Oct 28 21:35 db
-rw-r--r--    1 root     root        1.8K Oct 28 21:25 duc_startup.sh
-rw-r--r--    1 root     root        5.0M Oct 30 10:17 tmp.db

and this is the tkrzw output :

f5b996fee160:~# tkrzw_dbm_util inspect --dbm hash /duc/tmp.db
OpenAdvanced failed: NOT_IMPLEMENTED_ERROR: unsupported compression
f5b996fee160:~# tkrzw_dbm_util --version
Tkrzw utilities 1.0.32

To be honest it's a first for me compiling duc and especially it's tkrz dependency. I'vre read your doc and I would actually fit your description of "build the db somewhere and read it somewhere else" which would qualify me more for the sqlite db type, but the benefits of tkrzw seem great to me with faster run time given the many files I'm indexing... So getting tkrz & duc to work on alpine would be great as I'll be using the same alpine image everywhere to both build and display the indexes anyway.

Now to help you with docker to allow you helping me more, I have published the docker image to save you the trouble of actually building it and.
Duc is actually being build inside the container by the instructions in the dockerfile, I could have the dockerfile build the duc binary using ubuntu then run this binary inside an alpine container, I will try to see how I can do that and if it works better, I fear though it has to do with the tkrz lib and if there is a mismatch between what I build with and what I run with this also might be an issue...
To run it yourself and investigate you can :

1. install docker if you don't have it already : https://docs.docker.com/engine/install/debian/ (they have repos for other distros too)
2. pull the latest image I have build&published: docker pull toxic0berliner/duc-docker:alpine
3. instantiate the container with the above image and get a shell inside it: docker run --rm -it --hostname duc-docker toxic0berliner/duc-docker:alpine bash

in there you can do pretty much anything you like and you'll be in the exact same environment as my own development but also production containers 😄

Or you could build it yourself : _{(click to unfold more details)}

1. clone the repo git clone https://github.com/toxic0berliner/duc-docker.git
2. install docker if you don't have it already : https://docs.docker.com/engine/install/debian/ (they have repos for other distros too)
3. in the duc-docker folder, simply run docker build . which will run the Dockerfile and build an image layer by layer. I have updated the Dockerfile so it actually suceeds in building the image, so your build will end like this:

=> exporting to image                                                                                                                                                                 0.0s
=> => exporting layers                                                                                                                                                                0.0s
=> => writing image sha256:dfd62bf88a9d6b9fe7a978328daae0c99cb899fbe4bf8fc5f7d30a99e201650e                                                                                           0.0s

3 warnings found (use docker --debug to expand):
- LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 5)
- LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 64)
- JSONArgsRecommended: JSON arguments recommended for CMD to prevent unintended behavior related to OS signals (line 68

the sha256:df...50e part is what you are looking for: it's the id of the image

4. instantiate the container with the above sha256:df...50e image and get a shell inside it: docker run --rm -it --hostname duc-docker sha256:dfd62bf88a9d6b9fe7a978328daae0c99cb899fbe4bf8fc5f7d30a99e201650 bash

The Dockerfile in my repo is the build instructions executed by docker when building the image, it is stll a bit of a mess but still quite understandable so if you want to try other ways of doing things feel free to modify it or tell me anything you'd like to try I can attempt it 😄

which will allow you to update the `Dockerfile` allowing you to re-build duc or tkrz in any way you prefer. But of course I'll be happy with any help here and will try to aply modifications to my build script myself too !

[toxic0berliner]

I have tested building duc binary with debian:12 which works and copy the resulting binary into an alpine image, but all I get is this:

/usr/local/bin # ./duc
sh: ./duc: not found

The binary is there, it's executable, so it's most probably inside the binary that it's looking for something it doesn't find, so I think my best chance is most probably to build duc within alpine itself for it to find whatever it needs.

I did just realize that for debian I use the tkrzw from debian repo which seems to package something from https://dbmx.net/tkrzw/ which does seem the same source as I used but not the same version as it uses 1.0.25-1

Attempting another build with 1.0.25-1 instead... but sadly I still get the same error... build suceeds but indexing says unknown error.
If this can help here is the content of the db for an empty directory:

duc-builder:~# duc index /tmp/empty -d /duc/tmp.db
Error opening: /duc/tmp.db - Unknown error, contact the author
Unknown error, contact the author
duc-builder:~# cat /duc/tmp.db
TkrzwHDB
1P%TkrzwREC
1duc-builder:~

I'm open to any new idea, I'll be looking more closely at the duc build process now trying to find if there are any kind of errors I might have missed...

[l8gravely]

>>>> "toxic0berliner" == toxic0berliner ***@***.***> writes: I have tested building duc binary with debian:12 which works and copy the resulting binary into an alpine image, but all I get is this: /usr/local/bin # ./duc sh: ./duc: not found

What does 'ldd duc' say on both your debian build host and your alpine run host? And can you build duc on alpine? What version of alpine? I'll try to setup a copy of it here and test it out. If you can just setup a docker container with debian instead, that might fix all the issues.

The binary is there, it's executable, so it's most probably inside the binary that it's looking for something it doesn't find, so I think my best chance is most probably to build duc within alpine itself for it to find whatever it needs. I did just realize that for debian I use the tkrzw from debian repo which seems to package something from https://dbmx.net/tkrzw/ which does seem the same source as I used but not the same version as it uses 1.0.25-1

That is certainly something I would suspect to be a possible problem.

Attempting another build with 1.0.25-1 instead... but sadly I still get the same error... build suceeds but indexing says unknown error. If this can help here is the content of the db for an empty directory: duc-builder:~# duc index /tmp/empty -d /duc/tmp.db Error opening: /duc/tmp.db - Unknown error, contact the author Unknown error, contact the author duc-builder:~# cat /duc/tmp.db TkrzwHDB 1P%TkrzwREC 1duc-builder:~ I'm open to any new idea, I'll be looking more closely at the duc build process now trying to find if there are any kind of errors I might have missed...

Can you run duc properly on your build host? If so, then the problem is almost certainly a library mis-match between debian and alpine. And having now done a quick install of alpine linux 3.20 onto a VM, I think I see the problem. Alpine is a musl C library based system. Can you try doing the following inside an alpine linux container: $ su - # apk add gcompat # ./duc --version and see if that helps? I have never used alpine and just read a little on the wiki, but this jumped out at me. John

[l8gravely]

>>>> "toxic0berliner" == toxic0berliner ***@***.***> writes: Thanks a lot for your quick feedback ! Can you tell me which version of duc you're trying to use? [...] you're going to get a bunch of garbage when you use --version, which I need to fix. The version is indeed 1.5.0-rc1 but there is not much garbage in the output 😄 : f5b996fee160:~# duc --version Unknown option 'database' duc version: 1.5.0-rc1 options: cairo x11 ui tkrzw

This "Unknown option 'database' is coming from your ~/.ducrc file I bet. I'm chasing down this bug and trying to fix or update the message, since it's so distracting.

And since you're using tkrzw, could you use the utilities to check the file? Sure, I tried this to recreate the issue and get a db file that it claims it can't open. (click to unfold and see my exact commands to build this db & the error again)

So I think you will also need to copy a bunch of libraries to your alpine container, which might not be worth it. I'm playing around, but won't have any good solutions quickly here I'm afraid. Now that I've got an alpine VM running, I can do testing, but you will have to compile tkrzw on alpine so the libraries are there, and then probably compile duc as well on there. If you really need to use a container, use one that's the same os/arch as your main system for now. And if you do get it working on alpine, please writeup your instructions and I'll be happy to include them in the docs.

f5b996fee160:~# pwd /duc f5b996fee160:~# ls -alh ./ total 16K drwxrwxrwx 1 root root 4.0K Oct 28 21:35 . drwxr-xr-x 1 root root 4.0K Oct 30 10:11 .. drwxr-xr-x 2 root root 4.0K Oct 28 21:35 db -rw-r--r-- 1 root root 1.8K Oct 28 21:25 duc_startup.sh f5b996fee160:~# ls -alh /tmp total 8K drwxrwxrwt 1 root root 4.0K Oct 28 21:35 . drwxr-xr-x 1 root root 4.0K Oct 30 10:11 .. f5b996fee160:~# duc index /tmp -d ./tmp.db Error opening: ./tmp.db - Unknown error, contact the author Unknown error, contact the author f5b996fee160:~# ls -alh total 5M drwxrwxrwx 1 root root 4.0K Oct 30 10:17 . drwxr-xr-x 1 root root 4.0K Oct 30 10:11 .. drwxr-xr-x 2 root root 4.0K Oct 28 21:35 db -rw-r--r-- 1 root root 1.8K Oct 28 21:25 duc_startup.sh -rw-r--r-- 1 root root 5.0M Oct 30 10:17 tmp.db and this is the tkrzw output : f5b996fee160:~# tkrzw_dbm_util inspect --dbm hash /duc/tmp.db OpenAdvanced failed: NOT_IMPLEMENTED_ERROR: unsupported compression f5b996fee160:~# tkrzw_dbm_util --version Tkrzw utilities 1.0.32

So are you running 'tkrzw' on your alpine container (the f5b996fee160 above) or is this a debian container? If alpine, did you compile tkrzw yourself? In that case, you may need to also compile the zstd compression library on your alpine install.

To be honest it's a first for me compiling duc and especially it's tkrz dependency. I'vre read your doc and I would actually fit your description of "build the db somewhere and read it somewhere else" which would qualify me more for the sqlite db type, but the benefits of tkrzw seem great to me with faster run time given the many files I'm indexing... So getting tkrz & duc to work on alpine would be great as I'll be using the same alpine image everywhere to both build and display the indexes anyway.

Ok, so I think you'll need to do some work here, but I'll also see if I can find some time to build and test duc on alpine 3.20 as well. Hopefully shouldn't be too hard. I would recommend that you turn off all the CGI and GUI stuff with: ./configure --disable-cairo --disable-ui --disable-x11 --with-db-backend=tkrzw and see how that works for you for just indexing first.

Now to help you with docker to allow you helping me more, I am trying to publish the docker image publicly to save you the trouble of actually building it and I will be editing this message with the proper instructions once it's done.

I've now got an alpine VM running, quicker and easier than docker for me right now.

Duc is actually being build inside the container by the instructions in the dockerfile, I could have the dockerfile build the duc binary using ubuntu then run this binary inside an alpine container, I will try to see how I can do that and if it works better, I fear though it has to do with the tkrz[w] lib and if there is a mismatch between what I build with and what I run with this also might be an issue... But so far you can already:

Probably. The 'ldd duc' will tell us more.

1. clone the repo git clone https://github.com/toxic0berliner/duc-docker.git 2. install docker if you don't have it already : https://docs.docker.com/engine/install/debian/ (they have repos for other distros too) 3. in the duc-docker folder, simply run docker build . which will run the Dockerfile and build an image layer by layer. I have updated the Dockerfile so it actually suceeds in building the image, so your build will end like this: => exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:dfd62bf88a9d6b9fe7a978328daae0c99cb899fbe4bf8fc5f7d30a99e201650e 0.0s 3 warnings found (use docker --debug to expand): - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 5) - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 64) - JSONArgsRecommended: JSON arguments recommended for CMD to prevent unintended behavior related to OS signals (line 68 the sha256:df...50e part is what you are looking for: it's the id of the image 4. Instantiate the container with the above sha256:df...50e image and get a shell inside it: docker run --rm -it --hostname duc-docker sha256:dfd62bf88a9d6b9fe7a978328daae0c99cb899fbe4bf8fc5f7d30a99e201650 bash in there you can do pretty much anything you like and you'll be in the exact same environment as my own development but also production containers 😄

This all worked up until I needed to run it: $ docker run --rm -it --hostname duc-docker 66364e7871b4 /bin/sh: /duc/duc_startup.sh: Permission denied failed to resize tty, using default size So looking in assets/duc_startup.sh I see that it wasn't executable. So I did a chmod +x assets/duc_startup.sh and then 'docker build .' to update things. I'm clueless on docker. You should add the above instructions to the README.md in your git repo, and how to easily rebuild things as needed. Anyway, after that change and a rebuild, now I get: $ docker run --rm -it --hostname duc-docker 60e87dfac0eb /duc/duc_startup.sh: line 72: /usr/sbin/apache2ctl: No such file or directory failed to resize tty, using default size so I'm gonna punt working on this for now... can you give me hints on the best way to login and get a shell inside the container? Also, how do I tag a container properly and update instead of rebuilding? John

[toxic0berliner]

Wow, that's a lot of answers and a lot of effort you put in trying to help me, so first, thanks so much !

Making one thing crystal clear first : building duc inside docker based on a debian image or an ubuntu image runs flawlessly and I will publish a version of that dockerfile publicly too for others to use freely.
But as you can see the errors you're facing with the duc_startup and apache thing are some wrappers around your cgi output that I forked from another docker port that seems unmaintained, running on an old ubuntu image, and that I wanted to tweak anyways as I'll be building my custom version of duc where I added "dark mode", well, "low-effort" dark mode at least and rsized the graphs to 400px to fit on my phone screen 🤣 and that too is building and running fine inside a debian-based container today already, will publish that too shortly.
Anyway, those errors have nothing to do with duc and you could have picked an earlier layer to run, but hey, no matter, I'm gonna make it easier for you before the end of this message ;)

So don't sweat too much as what I'm now trying to do is make things "cleaner", using alpine is nothing I'm very familiar with but alpine is a very popular distro in docker world mainly because it's very small, and probably cause of the musl thing I'm unfamiliar with but which if I understand makes it lets say more "self contained". Not sure about that last part, but since it's so widely used, on a docker or kubernetes cluster there are some advantages to running several different containers based on the same base images, mostly space-wise, so this is the whole reason why I'm willing to have the alpine version as I never use alpine myself on my systems, I only consume many docker images that others build and are based on alpine mot often. So trying to follow the crowd mainly ;)

Let's try to address all your points now:
The version of alpine I used is in the Dockerfile : FROM: alpine:latest (the latest being implied), which as of now shows to be 3.20.3
ldd might be informative but is a foreign language to me: _{(not a C developer sorry and only sysadmin by night 😄 )}

duc-builder:~# ldd duc
/lib/ld-musl-x86_64.so.1: duc: Not a valid dynamic program

Now, after applying your advice of apk add gcompat and using your ./configure --disable-cairo --disable-ui --disable-x11 --with-db-backend=tkrzw the build suceeds and it looks different at least for ldd :

duc-builder:~# ldd /usr/local/bin/duc
        /lib/ld-musl-x86_64.so.1 (0x7fec84b17000)
        libtkrzw.so.1 => /usr/local/lib/libtkrzw.so.1 (0x7fec84920000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fec84b17000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x7fec84687000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fec84663000)

But, the duc index command still fails and tkrz_dbm_util inspect --dbm hash index.db also fails with the same unsupported compression message.

So I see you already have a quite close env to debug this yourself with an alpine VM, still I have tried to make docker easier for you as it seems you have tried to put yourself in my very shoes, so I did update my repo, you can git pull or take a new one, the README now show how to build :
Run docker build -t duc-docker . _{(the -t flag gives the image a user-defined name aka tag 😄)} then docker run --rm -it --hostname duc-docker duc-docker bash to get a shell in the container .
This last command will create a fresh container from the image every time and remove the container and everything you did inside it each time the bash command terminates.

But since I feel you come from the non-docker world, I believe you might prefer to do this instead:
docker run -it --hostname duc-builder --name duc-docker-debug duc-builder bash then exit out of the container.
Now you have an exited(stopped) container named duc-docker-debug that will remain there in the last state you left it until you explicitly delete it with docker rm duc-docker-debug.
An you can enter it at your leisure with docker start -ai duc-docker-debug, once you exit out of it it goes back to being stopped but keeping safe any changes or builds you might have done inside it, they will be right there were you left them when you next run docker start -ai duc-docker-debug

And lastly, to save you all the build time, you can instead use this
docker run -it --hostname duc-builder --name duc-docker-debug toxic0berliner/duc-docker:alpine bash, this will download from the image I built from hub.docker.com and let you run from the very same point as me.
Then exit out of this rist run command and use docker start -ai duc-docker-debug to start&enter the container again as you wish.
Whenever in the future I publish newer images with some progress, you will have to docker pull toxic0berliner/duc-docker:alpine to download the new one, docker rm duc-docker-debug to remove your stopped container and then reuse the docker run then docker start commands.

I have done some cleanup in the Dockerfile, you can have a look I have added the packages you advised (packages were available for gcompat, zstd, zstd-libs) they will be included in your build, even tried zstd-dev and zstd-static on my side with no more luck (but they are not included in the build yet you need to apk add them if you wish them)

Maybe I'm missing something else still..

Looking at the build process (with colors) I've spotted only a few warnings:

src/duc/cmd-histogram.c: In function 'histogram_db':
src/duc/cmd-histogram.c:47:38: warning: format '%d' expects argument of type 'int', but argument 4 has type 'size_t' {aka 'long unsigned int'} [-Wformat=]
   47 |                 printf("%3d %10s %'10d\n",i, pretty, count);
      |                                  ~~~~^               ~~~~~
      |                                      |               |
      |                                      int             size_t {aka long unsigned int}
      |                                  %'10ld
depbase=`echo src/duc/cmd-index.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
gcc -DHAVE_CONFIG_H -I.    -Isrc/libduc -Isrc/libduc-graph -Isrc/glad -g -O2 -MT src/duc/cmd-index.o -MD -MP -MF $depbase.Tpo -c -o src/duc/cmd-index.o src/duc/cmd-index.c &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo src/duc/cmd-info.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
gcc -DHAVE_CONFIG_H -I.    -Isrc/libduc -Isrc/libduc-graph -Isrc/glad -g -O2 -MT src/duc/cmd-info.o -MD -MP -MF $depbase.Tpo -c -o src/duc/cmd-info.o src/duc/cmd-info.c &&\
mv -f $depbase.Tpo $depbase.Po
src/duc/cmd-info.c: In function 'info_db':
src/duc/cmd-info.c:56:36: warning: '0' flag ignored with '-' flag in gnu_printf format [-Wformat=]
   56 |                             printf("2^%-02d  %'d\n",i, count);
      |                                    ^~~~~~~~~~~~~~~~
src/duc/cmd-info.c:56:48: warning: format '%d' expects argument of type 'int', but argument 3 has type 'size_t' {aka 'long unsigned int'} [-Wformat=]
   56 |                             printf("2^%-02d  %'d\n",i, count);
      |                                              ~~^       ~~~~~
      |                                                |       |
      |                                                int     size_t {aka long unsigned int}
      |                                              %'ld
depbase=`echo src/duc/cmd-ls.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\

which don't help me further sadly.

It feels like I'm simply missing a dependency again, especially given the message from tkrz_dbm_util, but can't figure out what... Since I feel the only compression is zstd I will attempt a manual build of zstd instead of using the packaged one, just to see if for some reason it's not finding it...

But for tonight I'm done, thanks again for all your help !

[l8gravely]

>>>> "toxic0berliner" == toxic0berliner ***@***.***> writes: Wow, that's a lot of answers and a lot of effort you put in trying to help me, so first, thanks so much !

You're welcome. Always happy to work with others, it helps keep me motivated to know others are using this.

Making one thing crystal clear first : building duc inside docker based on a debian image or an ubuntu image runs flawlessly and I will publish a version of that dockerfile publicly too for others to use freely.

Ok, so it's really an alpine issue, which I'm happy to try and help with, but is a bit outside my area of expertise. alpine uses a different C library called musl, instead of the linux standard glibc. So you need to compile on the same system type to make things work.

But as you can see the errors you're facing with the duc_startup and apache thing are some wrappers around your cgi output that I forked from another docker port that seems unmaintained, running on an old ubuntu image, and that I wanted to tweak anyways as I'll be building my custom version of duc where I added "dark mode", well, "low-effort" dark mode at least and rsized the graphs to 400px to fit on my phone screen 🤣 and that too is building and running fine inside a debian-based container today already, will publish that too shortly.

Yeah, those are docker issues which I won't bother with.

Anyway, those errors have nothing to do with duc and you could have picked an earlier layer to run, but hey, no matter, I'm gonna make it easier for you before the end of this message ;)

Yeah, it was interesting learning a bit more about Docker.

So don't sweat too much as what I'm now trying to do is make things "cleaner", using alpine is nothing I'm very familiar with but alpine is a very popular distro in docker world mainly because it's very small, and probably cause of the musl thing I'm unfamiliar with but which if I understand makes it lets say more "self contained". Not sure about that last part, but since it's so widely used, on a docker or kubernetes cluster there are some advantages to running several different containers based on the same base images, mostly space-wise, so this is the whole reason why I'm willing to have the alpine version as I never use alpine myself on my systems, I only consume many docker images that others build and are based on alpine mot often. So trying to follow the crowd mainly ;) Let's try to address all your points now: The version of alpine I used is in the Dockerfile : FROM: alpine:latest (the latest being implied), which as of now shows to be 3.20.3 ldd might be informative but is a foreign language to me: [(not a C developer sorry and only sysadmin by night 😄 )]

So ldd shows you which libraries are needed by an executable, and if it could find compatible ones.

duc-builder:~# ldd duc /lib/ld-musl-x86_64.so.1: duc: Not a valid dynamic program

Yeah, so this tells me your base OS C library isn't the same between compile hos and execution host.

Now, after applying your advice of apk add gcompat and using your ./configure --disable-cairo --disable-ui --disable-x11 --with-db-backend=tkrzw the build suceeds and it looks different at least for ldd : duc-builder:~# ldd /usr/local/bin/duc /lib/ld-musl-x86_64.so.1 (0x7fec84b17000)

This is the musl stuff.

libtkrzw.so.1 => /usr/local/lib/libtkrzw.so.1 (0x7fec84920000) libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fec84b17000) libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x7fec84687000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fec84663000)

This all looks ok. What does: ./duc --version show?

But, the duc index command still fails and tkrz_dbm_util inspect --dbm hash index.db also fails with the same unsupported compression message.

You might have to wait on my v1.5.0-rc2 branch to get pushed, I've got some tkrzw fixes to push and hopefully better detection and reporting of DB formats when you try to open a DB which you don't have support for. Supporting multiple DB backends at once would be a much bigger project and won't happen soon. Though it would be nice to have, but since they data does really change, it's just simpler to tell people to re-index and create a new DB.

So I see you already have a quite close env to debug this yourself with an alpine VM, still I have tried to make docker easier for you as it seems you have tried to put yourself in my very shoes, so I did update my repo, you can git pull or take a new one, the README now show how to build : Run docker build -t duc-docker . [(the -t flag gives the image a user-defined name aka tag 😄)] then docker run --rm -it --hostname duc-docker duc-docker bash to get a shell in the container . This last command will create a fresh container from the image every time and remove the container and everything you did inside it each time the bash command terminates. But since I feel you come from the non-docker world, I believe you might prefer to do this instead: docker run -it --hostname duc-builder --name duc-docker-debug duc-builder bash then exit out of the container. Now you have an exited(stopped) container named duc-docker-debug that will remain there in the last state you left it until you explicitly delete it with docker rm duc-docker-debug. An you can enter it at your leisure with docker start -ai duc-docker-debug, once you exit out of it it goes back to being stopped but keeping safe any changes or builds you might have done inside it, they will be right there were you left them when you next run docker start -ai duc-docker-debug And lastly, to save you all the build time, you can instead use this docker run -it --hostname duc-builder --name duc-docker-debug toxic0berliner/duc-docker:alpine bash, this will download from the image I built from hub.docker.com and let you run from the very same point as me. Then exit out of this rist run command and use docker start -ai duc-docker-debug to start&enter the container again as you wish. Whenever in the future I publish newer images with some progress, you will have to docker pull toxic0berliner/duc-docker:alpine to download the new one, docker rm duc-docker-debug to remove your stopped container and then reuse the docker run then docker start commands. I have done some cleanup in the Dockerfile, you can have a look I have added the packages you advised (packages were available for gcompat, zstd, zstd-libs) they will be included in your build, even tried zstd-dev and zstd-static on my side with no more luck (but they are not included in the build yet you need to apk add them if you wish them)

Thanks for all these instructions, they will help when I get to that level of testing. For now I'm just trying to get a working build in alpine.

Maybe I'm missing something else still.. Looking at the build process (with colors) I've spotted only a few warnings: src/duc/cmd-histogram.c: In function 'histogram_db': src/duc/cmd-histogram.c:47:38: warning: format '%d' expects argument of type 'int', but argument 4 has type 'size_t' {aka 'long unsigned int'} [-Wformat=] 47 | printf("%3d %10s %'10d\n",i, pretty, count); | ~~~~^ ~~~~~ | | | | int size_t {aka long unsigned int} | %'10ld depbase=`echo src/duc/cmd-index.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\ gcc -DHAVE_CONFIG_H -I. -Isrc/libduc -Isrc/libduc-graph -Isrc/glad -g -O2 -MT src/duc/cmd-index.o -MD -MP -MF $depbase.Tpo -c -o src/duc/cmd-index.o src/duc/cmd-index.c &&\ mv -f $depbase.Tpo $depbase.Po depbase=`echo src/duc/cmd-info.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\ gcc -DHAVE_CONFIG_H -I. -Isrc/libduc -Isrc/libduc-graph -Isrc/glad -g -O2 -MT src/duc/cmd-info.o -MD -MP -MF $depbase.Tpo -c -o src/duc/cmd-info.o src/duc/cmd-info.c &&\ mv -f $depbase.Tpo $depbase.Po src/duc/cmd-info.c: In function 'info_db': src/duc/cmd-info.c:56:36: warning: '0' flag ignored with '-' flag in gnu_printf format [-Wformat=] 56 | printf("2^%-02d %'d\n",i, count); | ^~~~~~~~~~~~~~~~ src/duc/cmd-info.c:56:48: warning: format '%d' expects argument of type 'int', but argument 3 has type 'size_t' {aka 'long unsigned int'} [-Wformat=] 56 | printf("2^%-02d %'d\n",i, count); | ~~^ ~~~~~ | | | | int size_t {aka long unsigned int} | %'ld depbase=`echo src/duc/cmd-ls.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\ which don't help me further sadly. It feels like I'm simply missing a dependency again, especially given the message from tkrz_dbm_util, but can't figure out what... Since I feel the only compression is zstd I will attempt a manual build of zstd instead of using the packaged one, just to see if for some reason it's not finding it... But for tonight I'm done, thanks again for all your help ! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.*Message ID: ***@***.***

com>

[l8gravely]

Quick note, when you compile tkrzw on alpine, you need do: ./configure --with-zstd to make sure it compiles in support for zstd library. Which means you might need to do: apk add zstd zstd-dev as well. Still testing.

[toxic0berliner]

Alredy had zstd, added zstd-dev and it turns out for tkrzw 1.0.25 the flag is --enable-zstd but still no luck.
Which version of tkrzw would you advise? 1.0.25 complained about the flag you mentioned... In the dockerfile I'm preparing I am downloading and building tkrzw anyway, I can either pin it to a version of our choosing or pull from their master branch on GitHub.

[l8gravely]

>>>> "toxic0berliner" == toxic0berliner ***@***.***> writes: Alredy had zstd, added zstd-dev and it turns out for tkrzw 1.0.25 the flag is --enable-zstd but still no luck.

Me either... but haven't had time to spend on it today much. Will do more checks this weekend.

Which version of tkrzw would you advise? 1.0.25 complained about the flag you mentioned... In the dockerfile I'm preparing I am downloading and building tkrzw anyway, I can either pin it to a version of our choosing or pull from their master branch on GitHub.

I'd pull down the latest version, but getting it to compile with ZSTD seems to be the problem on alpine right now. More when I dig deeper.

[toxic0berliner]

Thanks in advance! And again, don't apologize or rush it there is no emergency and I myself have family over for the weekend and limited availability. I probably won't make it this weekend but my current plan was to find out which flags they use to build tkrzw for the Debian repos and try to use the same since it worked fine building on Debian using their packaged libs

[l8gravely]

>>>> "toxic0berliner" == toxic0berliner ***@***.***> writes: Thanks in advance! And again, don't apologize or rush it there is no emergency and I myself have family over for the weekend and limited availability. I probably won't make it this weekend but my current plan was to find out which flags they use to build tkrzw for the Debian repos and try to use the same

I've made some progress here, but I want to put together a proper set of patches for v1.5.0-rc2 to fix this. Basically, when tkrzw is compiled, it defaults back to lz4 compression (if found I think), and the docker config doesn't support that. So the answer is to: 1. update duc's ./configure to detect this and properly warn about it. 2. compile tkrzw with: apk add liblzma-dev apk add xz-dev apk add xz-libs apk add lz4-dev ... all the other stuff already added ... ./configure --enable-lz4 --enable-zstd --enable-zlib --enable-lzma Once I do the above (with some hacks) I can now index and view files. I've also got to fix the ncurses printw() call in cmd-ui.c to have the right arg types, and all my committed patches are really quite wrong so I need to revert some stuff or combine them down first. But you can try looking at the v1.5.0-rc2 branch and trying it out if you like. No promises though!

[toxic0berliner]

thanks for the feedback and sorry for the delay.
Seems There is no "liblzma-dev" package for alpine, so I removed it as well from the tkrzw configure options.
I switched to v1.5.0-rc2 but it fails to build claiming libarchive is not found. It was not previously showing as missing, I added it with apk nonetheless but still no luck.
I've become a father very recently so time is becoming limited on my side sadly, I will try to dig into it a bit more when I can.
Thanks again for all the help already !

[l8gravely]

>>>> "toxic0berliner" == toxic0berliner ***@***.***> writes: thanks for the feedback and sorry for the delay.

Sorry for my delay as well! Thanksgiving has slowed me down for sure...

Seems There is no "liblzma-dev" package for alpine, so I removed it as well from the tkrzw configure options.

Hmmm. which version of Alpine are you running? I've got a 3.20 test instance running.

I switched to v1.5.0-rc2 but it fails to build claiming libarchive is not found. It was not previously showing as missing, I added it with apk nonetheless but still no luck.

Please share the config.log and the build log, that will help me debug this.

I've become a father very recently so time is becoming limited on my side sadly, I will try to dig into it a bit more when I can. Thanks again for all the help already !

Congratulations! As an older dad, let me tell you that diapers aren't nearly the big deal I thought they would be. :-)

[l8gravely]

>>>> "toxic0berliner" == toxic0berliner ***@***.***> writes: Seems There is no "liblzma-dev" package for alpine, so I removed it as well from the tkrzw configure options.

Ah, try using the following: # apk search liblzma xz-dev-5.6.2-r0 xz-libs-5.6.2-r0 # apk install xz-dev xz-libs

I switched to v1.5.0-rc2 but it fails to build claiming libarchive is not found. It was not previously showing as missing, I added it with apk nonetheless but still no luck.

Hmm... I just did a configure && compile and I've got libarchive installed on here. More details when you get a chance! John