C scanner misses indirect dependencies on generated header files

[bdbaddog]

This issue was originally created at: 2012-06-27 08:01:20.
This issue was reported by: gward.

gward said at 2012-06-27 08:01:20

I have a trivial project with three source files:

  app/file1.c
  app/file2.c
  lib1/lib1.h

each of which is also trivial:

$ cat app/file1.c
#include <lib1.h>
 
$ cat app/file2.c
#include <zlib.h>
 
$ cat lib1/lib1.h 
#include <zlib.h>

The catch: zlib.h is "generated" by copying it from /usr/include to a local include directory. (This is a simplified version of our production system, which downloads and unpacks dozens of third-party C, C++, and Java libraries and tools as part of the build process. We do it that way to ensure strict control over all third-party dependencies.)

Thus, the build process should look like this:

# read list of third-party libs from libs.txt and copy headers
# for those libs from system /usr/include to local include/
./copyheaders libs.txt include

gcc -o app/file1.o -c -Iinclude -Ilib1 app/file1.c
gcc -o app/file2.o -c -Iinclude -Ilib1 app/file2.c

In fact, that was pretty easy to setup. But if I do things the straightforward way, I always get unnecessary rebuilds:

scons: rebuilding `app/file1.o' because:
           `include/zlib.h' is a new dependency
           `include/zconf.h' is a new dependency
gcc -o app/file1.o -c -Iinclude -Ilib1 app/file1.c
scons: rebuilding `app/file2.o' because:
           `include/zlib.h' is a new dependency
           `include/zconf.h' is a new dependency
gcc -o app/file2.o -c -Iinclude -Ilib1 app/file2.c

So I'm using a clever trick based on http://scons.org/wiki/DynamicSourceGenerator:

• copyheaders writes a file (include/.contents) listing all of the header files that it copied to include/
• custom builder that reads include/.contents and adds those filenames to the dependency graph -- note that this is adding dependencies during the build phase, which is a bit unorthodox but sometimes works

This clever trick works... for file2.o, but not for file1.o. Specifically, I can fix the unnecessary rebuilds of file2.o, but I still get unnecessary rebuilds of file1.o. Recall that the only difference is that file2.c includes <zlib.h> directly, whereas file1.o includes it via <lib1/lib1.h>.

I can demonstrate the bug by doing a build from a clean working dir:

$ scons --debug=explain --tree=prune
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: building `include/.contents' because it doesn't exist
./copyheaders libs.txt include
`/usr/include/zconf.h' -> `include/zconf.h'
`/usr/include/zlib.h' -> `include/zlib.h'
scons: building `dummy-include' because it doesn't exist
ReadOutputs(["dummy-include"], ["include/.contents"])
reading include/.contents
done
scons: building `app/file1.o' because it doesn't exist
gcc -o app/file1.o -c -Iinclude -Ilib1 app/file1.c
scons: building `app/file2.o' because it doesn't exist
gcc -o app/file2.o -c -Iinclude -Ilib1 app/file2.c
+-.
  +-SConstruct
  +-app
  | +-app/file1.c
  | +-app/file1.o
  | | +-app/file1.c
  | | +-dummy-include
  | | | +-include/.contents
  | | |   +-copyheaders
  | | |   +-libs.txt
  | | +-lib1/lib1.h
  | | +-/usr/bin/gcc
  | +-app/file2.c
  | +-app/file2.o
  |   +-app/file2.c
  |   +-[dummy-include]
  |   +-include/zlib.h
  |   | +-[dummy-include]
  |   +-include/zconf.h
  |   | +-[dummy-include]
  |   +-/usr/bin/gcc
  +-copyheaders
  +-[dummy-include]
  +-include
  | +-[include/.contents]
  | +-[include/zconf.h]
  | +-[include/zlib.h]
  +-lib1
  | +-lib1/lib1.h
  +-libs.txt
scons: done building targets.

It's right there in the graph: file2.o depends on include/zlib.h (correct), but file1.o does not (incorrect). The consequence: a second run of scons incorrectly rebuilds file1.o

$ scons --debug=explain --tree=prune
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: building `dummy-include' because it doesn't exist
ReadOutputs(["dummy-include"], ["include/.contents"])
reading include/.contents
done
scons: rebuilding `app/file1.o' because:
           `include/zlib.h' is a new dependency
           `include/zconf.h' is a new dependency
gcc -o app/file1.o -c -Iinclude -Ilib1 app/file1.c
+-.
  +-SConstruct
  +-app
  | +-app/file1.c
  | +-app/file1.o
  | | +-app/file1.c
  | | +-dummy-include
  | | | +-include/.contents
  | | |   +-copyheaders
  | | |   +-libs.txt
  | | +-lib1/lib1.h
  | | +-include/zlib.h
  | | | +-[dummy-include]
  | | +-include/zconf.h
  | | | +-[dummy-include]
  | | +-/usr/bin/gcc
  | +-app/file2.c
  | +-app/file2.o
  |   +-app/file2.c
  |   +-[dummy-include]
  |   +-[include/zlib.h]
  |   +-[include/zconf.h]
  |   +-/usr/bin/gcc
  +-copyheaders
  +-[dummy-include]
  +-include
  | +-[include/.contents]
  | +-[include/zconf.h]
  | +-[include/zlib.h]
  +-lib1
  | +-lib1/lib1.h
  +-libs.txt
scons: done building targets.

Note that the graph is correct now, so we don't get any more unnecessary rebuilds.

You can get my example project here:

hg clone http://hg.gerg.ca/copyheaders2/

or

http://hg.gerg.ca/copyheaders2/archive/tip.tar.gz

dirkbaechle said at 2014-05-18 03:10:53

reassigning issue

[bdbaddog]

@gward - Is this bug still and issue? It's possible it's been fixed in recent releases. Also if not is your test case available elsewhere?