Multiple Executables in Project

[cwfitzgerald]

My use case is as follows: I have a selection of library based submodules. They are not completely standalone as there are dependencies among them, but they makes sense as submodules. I also will be generating multiple executables that use these libraries to do various things. There doesn't seem to be a way in the current spec to have "submodules" in the src directory. It makes logical sense to me that you put library submodules in the libs directory and executable submodules in the src directory.

Random illustrative example: if my project generated: libdisk.so libnetwork.so disk-add network-add the structure would be like:

|-- lib
|   \-- disk
|       \-- disk.cpp
|   \-- network
|       \-- network.cpp
|-- src
    \-- disk-add
        \-- main.cpp
    \-- network-add
        \-- main.cpp

Thoughts?

[SGSSGene]

I agree with this. But I think it is semantically not sound. Why are some sources in "lib" and others in "src"?
So I think the next logical step would be to move everything into src and let a outside tool decide what is a library or executable.

 -- src
    \-- disk
        \-- disk.cpp
    \-- disk-add
         \-- main.cpp
    \-- network
        \-- network.cpp
     \-- network-add
         \-- main.cpp

But if we already do this, we should also consider unit tests written in c++. These actually could also be located here:

 -- src
    \-- disk
        \-- disk.cpp
    \-- disk-add
        \-- main.cpp
    \-- network
        \-- network.cpp
    \-- network-add
        \-- main.cpp
    \-- test-disk
        \-- main.cpp
    \-- test-network
        \-- main.cpp

Interestingly the issue of deciding what is a library and what an executable is now very easy. Some where the dependencies have to be encoded. No sub-modules are depending on executables (apps and tests) which makes it easy to detect. Every library should have at least one unittest so they do have other submodules depending on them. This way libraries can be easily identified by a build system.
And it would make the whole setup easier and closer to a "zero configurable" solution.

[DethRaid]

Do you only have one main executable, and a few utility type things that aren't a main program?

I'd put the libraries in libs, the main executable in src, and the utilities in extras, each within their own folder. Imo this has the best semantics and best implements the Pitchfork convention

[cwfitzgerald]

Note

I made a mistake in my directory diagram. I didn't include the proper subdirectories in submodules. I also wrote lib over libs (also making ascii charts right is hard and I missed some edges). Fixed diagram:

\-- libs
|   \-- disk
|   |   \-- include
|   |   |   \-- disk.hpp
|   |   \-- src
|   |       \-- disk.cpp
|   \-- network
|       \-- include
|       |   \-- network.hpp
|       \-- src
|           \-- network.cpp
\-- src
    \-- disk-add
    |   \-- src
    |       \-- main.cpp
    \-- network-add
        \-- src
            \-- main.cpp

Response @SGSSGene

I disagree. I think it is important for the directory structure to contain distinctions between the various components. The entire point of having a separation between include, src, and tests within any one submodule is that while they may all be written in the same language, they have very different purposes and styles. Your include directory is meant to be used externally, and is mainly forward decls. Your source is where you define those things as is meant mainly to be internal only (besides skimming the source if you think you've found a bug). Your tests are meant for your use only. They are there to verify the correctness of the library, not actually contribute to the external interface of your project.

Why are some sources in "lib" and others in "src"?

For the same reason each submodule has three subdirectories for different types of c++ code. The type of code you write in a library is often very different then the code you write for an executable. The library needs to be generic, often needs to follow API and ABI stability rules, and often contains many indirections for user customizability. Executables on the other hand will never have their code reused, so their code is often much more concrete. They are tying all the library code together to make a directly usable program that allows you to have a human acceptable interface to the libraries behind it. This code is often very different to the library code. For this reason it should be separate. Similarly it just doesn't make sense to have executable code in libs or vice versa.

Interestingly the issue of deciding what is a library and what an executable is now very easy. Some where the dependencies have to be encoded. No sub-modules are depending on executables (apps and tests) which makes it easy to detect.

This makes it easy for some kind of build-system aware tool to detect which is which, but it makes it very hard for humans to determine between them just by looking at structure alone. Similarly, what about libraries that only exist to be used elsewhere and don't (or can't) have unit tests. There you have a library that is detected as an executable. While on a complete project with unit tests and all that heuristic works, but it's just that, a heuristic.

And it would make the whole setup easier and closer to a "zero configurable" solution.

If we have this idea that each submodule can have their own directory tree and you can have submodules in extra, libs, and src this makes it very easy to figure out what is what. Mark your project as submodule aware and a tool can determine which libraries exist through enumerating libs, which extra executables exist by enumerating extra, and which executables exist by enumerating src (I think it would be useful to have a libs/<module>/extra/ to contain extra libraries directly relating to a submodule (such as their language bindings) but that's another discussion).

Response @DethRaid

I have more than one executable that it flat out doesn't make sense to build without. I need to interface with a 32bit or CLR library as a 64bit program and the only real way to do that is through IPC. Luckily the calls are light enough that that is viable. So compiling without this program doesn't make sense as the libraries I need to talk to won't work.

[DethRaid]

In that case, I agree about putting multiple executables in src/. I'd still put as much as possible in extras/, but if the executable is needed for your main program to run then yeah make submodules in src/

[amaiorano]

There doesn't seem to be a way in the current spec to have "submodules" in the src directory.

In fact, the way I read it, the spec says that if you use submodules, you must not have a root src (nor include) at all, only libs. So executables as well as lib submodules all end up under the libs/ root folder. I find this weird as well, and do like what you propose.

[SGSSGene]

Thank you for your great response!

I misunderstood your diagram completely. I think you are correct. You could checkout the program "tree" helping to make those file diagrams. It is a little like "ls" but outputs a tree.

Maybe submodules is not the right tool for you. Maybe your layout should look more like this:

<root>
├── apps
│   ├── disk-add (pf root)
│   │   └── src
│   │       └── main.cpp
│   └── network-add (pf root)
│       └── src
│           └── main.cpp
└── libs
    ├── disk (pf root)
    │   ├── include
    │   │   └── disk.h
    │   └── src
    │       └── disk.cpp
    └── network (pf root)
        ├── include
        │   └── network.h
        └── src
            └── network.cpp

Notice that in this setup there exists 4 pitchfork layout root folders. Every library and executable has there own.

[cwfitzgerald]

That seems to defeat the purpose of the proposal to me. The main point is that your project is not special and does not need a custom directory structure. Having your repo root be anything other than a pf root seems to be treating it as special.

I would be okay, however, with apps becoming like libs but for executables. It is a submodule root, and must not exist when src exists. This would be a clear way to allow multiple excutables while allowing src to a component of a single submodule only.

re: tree I thought about it, but I thought that actually making the directory tree would be too much work, but in the end the ASCII chart was harder and I made a bunch of mistakes on it. 🤷‍♂️

[Quincunx271]

I believe the submodules layout is intended only for libraries:

Its presence excludes the src/ and include/ directories.

Note what vector-of-bool states in the section on submodules:

Note: Splitting a project into submodules should be considered very carefully. It is an extremely heavy tool with subtleties that often trip people up. Very few projects warrant subdividing themselves in this way. Most projects will do just fine with multiple namespaces and directories within their source tree. Don’t reach for this tool when namespaces and subdirectories will suite you just fine. Converting a project to/from a submodule layout is a very cumbersome task.

Then also:

The following rules must be taken into consideration when considering or using submodules:

• Submodules are not themselves standalone projects.
• They should not pretend to be entire projects.
• They cannot be consumed independent of the rest of the project.
• They should not be versioned separately from the project.
• They cannot further subdivide into sub-submodules.

I take this to mean: if you're producing a standalone executable in a submodules fashion, that goes against these rules. Instead, you must split the project into pieces, or not use the submodules layout.

---

If you want to have an executable which depends on a submodule-layout library, you could create a separate project. This can be done with an entirely new repo, or you could do pretty much what SGSSGene is suggesting and host several projects in the same repo.

If the attached executables are very useful / tightly coupled to the library, you could consider putting them in extras/.

---

I wouldn't do this with a submodule layout. Instead, I'd do something like this:

+- include/
|  +- my_tool/
|     +- disk.hpp
|     +- network.hpp
|
+- src/
   +- my_tool/
   |   +- disk.cpp
   |   +- network.cpp
   |
   +- disk-add.cpp
   +- network-add.cpp

As vector-of-bool says, "Very few projects warrant subdividing themselves [into submodules]."

[cwfitzgerald]

Some Context

I've been going through this being a little vague about what I'm doing, so I think it would be helpful to bring in my actual full context in order to better illustrate why I have the opinion I do. My project is BVE-Reborn (github mirror). It is a remake of a already existing game. The project has two goals: have a better architecture, and to allow the building of tools for the game. Because of both goals, there is a need to split up the code into smaller libraries with a single set of concerns, so I can write tools that will utilize that code to do things.

Due to legacy compatibility issues (I need to talk to a 32bit (or .NET) plugin dll from a 64bit native project), I need to produce, at bare minimum, the game executable and the helper executable. The game talks to the helper through IPC, basically doing RPC instead of directly calling the dll.

As of right now the project isn't massive in terms of files or lines of code, 16k SLOC over 175 files, but when the project is taken to completion, it is going to end up quite large. With so many different parts, splitting it up into submodules makes sense.

Response

I believe the submodules layout is intended only for libraries

I know, and that's what I want to change 😄

I take this to mean: if you're producing a standalone executable in a submodules fashion, that goes against these rules. Instead, you must split the project into pieces, or not use the submodules layout.

While they may be "standalone" in the sense that they are both executables, the project cannot work correctly if both are not present. The game relies on the helper executable to exist to properly talk with the plugins that are needed to give the proper behavior. The way I take that quote to mean is that you can't have two orthogonal projects within the same tree and just call them submodules. In my case, all parts are intimately connected with each other. They all have non-circular dependencies on each other and all evolve in complete lockstep.

If the attached executables are very useful / tightly coupled to the library, you could consider putting them in extras/.

I would be willing to do this for the tools that I build as they not truly necessary, and the game and the helper don't depend on any of them.

This can be done with an entirely new repo, or you could do pretty much what SGSSGene is suggesting and host several projects in the same repo.

Again, this seems to go against the idea that "your project isn't special, everyone should use this one format". Adding multiple pf roots is doing exactly that, special-casing your project.

This issue is intended to find some kind of official solution to the problem. In the mean time I have adopted a solution that tries to remain as close as possible to pitchfork while keeping the folder structures I need. The rules for that are as follows:

• extras, libs, src, and tests are submodule roots
  • extras is for non-mandatory executables/libraries
  • libs is for mandatory libraries
  • src is for mandatory executables
  • tests is for executables and libraries only related to testing but can't be part of the library's tests subdirectory.
• All top-level library submodules are allowed to have extra as a nested submodule root. This allows things like language bindings and other intimately related things to be kept in the same submodule. This should not be used for large things, or things that are not intimately tied to the library they are in. I have an example of this in libs/bve-core/extras/swig

I know I probably won't be able to get this exact format into the spec, but this seems to keep with the intent of the proposal while allowing more flexibility.

I wouldn't do this with a submodule layout.

For a project of that size, and that scope, I completely agree. I should have been more clear though that I was using that as an example.

[Quincunx271]

I had a quick look, and I'd still do it with the non-submodule layout. I do have large projects in the non-submodule layout which do have sections that are largely independent from other sections, but the non-submodule layout works great for me.

However, I can see the point in your structure. The more I think about it, the more it makes sense.

---

Not really important because it wouldn't be a good fit for your project as you really do have submodules and not separate projects, but I would argue that having multiple pf roots in a repo isn't "your project is special", because it would be a collection of projects in a monorepo. Doesn't apply for this particular case, though.

[poconbhui]

I've found a .main.cpp suffix to be useful. So my_executable.main.cpp gets compiled to my_executable, or me_executable.exe. The main function is a significant enough addition to a translation unit that I find it's worth noting in the name.

e.g. taking the example from @Quincunx271

+- include/
|  +- my_tool/
|     +- disk.hpp
|     +- network.hpp
|
+- src/
   +- my_tool/
   |   +- disk.cpp
   |   +- network.cpp
   |
   +- disk-add.main.cpp
   +- network-add.main.cpp

makes the my_tool library, and the disk-add and network-add executables.

Similarly, you might have

+- include/
|  +- my_tool/
|     +- disk.hpp
|     +- network.hpp
|     +- my_tool.hpp
|
+- src/
   +- my_tool/
   |   +- disk.cpp
   |   +- network.cpp
   |   +- my_tool.cpp
   |   +- my_tool.main.cpp

where the bulk of the main body is implemented in my_tool.cpp, with my_tool.hpp providing the interfaces needed to wrap them in a main function. This can be useful for unit testing a main function, and for users to implement modified executables of your project.

[vector-of-bool]

Hello all! I've been on a bit of hiatus, but hoping to get back into the swing of things.

I just got caught up on this ticket, and I'd say that having (multiple) executables is still one of the biggest open design questions. PFL is currently very specialized for libraries and needs more provisions specifically for executables.

I've been toying around with a few ideas, and it seems that some folks in this thread have "independently discovered" the same ideas. There are a few needs to consider:

1. Many projects are not themselves libraries and are not intended to be consumed as such. Their primary output are executables, which may be better described as "applications" to distinguish from intermediate executables and tests. In general I've come to refer to them as "application" projects (Chromium, Apache HTTPd).
2. Many projects do not expose applications/executables and are just "library" projects (Boost).
3. Some projects expose both libraries and applications (LLVM, Qt). I'm not sure how to name them, perhaps "hybrid" projects?

"Hybrid" projects are often the most troublesome in terms of finding a solid solution, but this may also be a concern of application projects which produce more than a single application.

One of the ideas kept in mind during designing PFL was that, other than the main entrypoint, it is desirable to "library-ify" applications such that main() is essentially a trampoline into the application's "library." This is especially useful if multiple entrypoints share the vast majority of their backend code (e.g. A graphical application and a command-line application that both act as interfaces to the same underlying task). Even if a project does not expose a library as a final product, having a "backend" library for the application is an excellent aide for testing and extensibility.

I want to distinguish an "entrypoint" from main(). While main()/wmain()/WinMain()/etc. is the technical entrypoint of a program, there is often more code than just this entrypoint function which makes up the application's "entrypoint" interface.

It may be useful to think of all code except the application's "library backend" as the entrypoint of an application. An entrypoint may consist of a single main() that accepts no command-line arguments, or it could be a vast number of dialogs and visual prompts that are presented to the user. It may be that the entrypoint is a GUI that continually invokes the backend library and presents the results to the user. "Entrypoint" is starting to become a bit of a flimsy term here, but, for lack of a better term, I'll continue using it.

Simple Entrypoints: Using a .main.cpp Suffix

For simple entrypoints -- especially command line applications -- it is simple enough to provide a single source file that provides an entrypoint and maps the command-line-arguments/environment/input into calls into the "backend" library. Such a design is satisfied by @poconbhui's offering above, and using a .main.cpp suffix to denote this "entrypoint" source file is one of the options I considered but never wrote down formally.

Another option I considered early on was distinguishing these lone entrypoint source files by placing them at the root of src/ (with no qualified path), and using the filename stem as the name of the generated executable. This could work, but may be confusing.

This simple design will work for both application projects and hybrid projects, as there will be no confusion about what sources denote the "backend" library and which sources denote the entrypoints.

This also works well for projects that expose multiple entrypoints, as a single source .main.cpp file can correspond to a single entrypoint. Huzzah!

Things can get a little hairier for complex projects, though...

Complex Entrypoints: Using a .main.cpp Suffix ?

For graphical applications and intricate command-line applications it is very common (and often necessary) to segment the interface into distinct components and giving those components their own source files as their complexity grows. Consider the complexity of source control applications like Git, Subversion, or Mercurial, which all have many sub-commands. It would be frightening if the entirety of Git's command-line were written into a single git.main.c. We'll want to split the entrypoint to better understand our project.

Despite these splits, we will probably share a lot of code between our entrypoints and entrypoint components. Consider shared GUI widgets or common command-line argument parsing code.

For application projects (which do not expose a library as a final product), using .main.cpp may be sufficient, as the entrypoint code and backend code can live together [in harmony until the fire nation attacked].

However: We're now breaking one of the goals of "library-ifying" an application: The entrypoint and backend code is all mixed together! While you and I might have the discipline to keep this sorted, newcomers and potential contributors might not. (And I'm starting to doubt that I will, come five months down the road).

Mixing complex entrypoint and backend code together in a single source directory also opens up the possibility (and temptation) to #include a file from the entrypoint into the backend. At that point, all bets are off. Human sacrifice, dogs and cats living together, mass hysteria!

Complex Entrypoints: Using libs/ ?

As has been discussed, we might use libs/ to create "entrypoint" submodules to segment our application code from the backend code. This is a completely viable option, but it does seem weird to put code for an executable into a directory named "libs." As @Quincunx271 says, it would makes sense of "libs" to be only for libraries.

It also may be a bit too powerful for the task. We may want multiple executables, but we don't want to be required to suddenly split our program into a dozen different subdirectories.

Complex Entrypoints: Using apps/

Having another top-level directory named apps/ is another solution I thought about, and @SGSSGene and @cwfitzgerald seem to be on the same track.

A benefit of apps/ is that it can act as another "source directory" and #include root dedicated to entrypoint code. This prevents the mixing of entrypoint and backend code, and allows us to quickly understand the structure of a more complex application.

While "your project is probably not special" may be true, there seem to be a lot of "special" projects that could benefit from an apps/ directory.

An open question is "What should apps/ look like?" I'm tempted to refrain from definitively ruling it to be a submodule root exclusively, as we start to end up with a "directory soup" between apps/, libs/, extra/, etc. I'm leaning toward making it an "either-or" between being a source directory like src/ and a submodule root like libs/, and it would be up to the project to pick which structure it would benefit from most.

For projects with a single entrypoint, treating it as a source directory would be just fine, as the additional level of indirection would be unecessary. For projects with multiple entrypoints, treating it as a submodule root with one submodule per entrypoint may be prudent.

Another option could be a hybrid: Treat apps/ as a single source directory with a single "entrypoint library", which each main() contained in a .main.cpp file. At this point we might be getting excessively granular.

Proposed Changes:

I think the best option is a hybrid approach, which requires some care. While I would like to mandate the "library-ification" of applications, such a mandate is difficult to encode and would be likely very divisive. Rather, it may be best to provide the necessary provisions to make it a very attractive option, and provide a uniform design such that it is easily understood where the division between "entrypoint" and "backend" reside.

Considering all of the above, and the discussion in this thread, here is my initial proposition, open to feedback of course:

1. Formally distinguish "entrypoint" code as code which is directly executed and driven by an end-user not via a native API. The distinction between backend and entrypoint code is mostly conceptual and is meant as an aide to human readers.

2. Formally distinguish an "application entrypoint function" as main()/wmain()/WinMain()/`etc. and designate the source file that contains this function to be the "application entrypoint function source file." [A bit of a mouthful. Feedback wanted.]

3. Denote that any file with a .main.* suffix to be an application entrypoint function source file, and it must contain an application entrypoint function.

4. For any application entrypoint function source file placed in a source directory src/ (be that at the top-level or within a submodule), that source file shall contain only entrypoint code, and all other source files within that source directory will contain "backend" code.

   <root-or-submodule>
   │
   └─ src/
      ├ something.main.cpp  <- Generates something.exe
      ├ meow.main.cpp       <- Generates meow.exe
      └ my_lib/             <- "backend" code
        ├ my_lib.hpp
        └ my_lib.cpp
   

5. Add a new directory which may appear at the top-level or within a submodule root: apps/, designed to hold application entrypoint code. There are three options for layout of apps/, each designed for a different size and complexity of project.

   1. The "simple" layout: apps/ may contain any number of compilable source files at its root. Each source file is an application entrypoint function source file, and should contain an application entrypoint function. Each compilable source file in a simple layout will generate a different executable. Compilation of files in apps/ should link and #include the content of its sibling src/ directory. This is for applications that have multiple entrypoints but otherwise all significant functionality is defined within its "backend" library contained in src/.

   <root-or-submodule>
   │
   ├─ apps/
   │  ├ foo.cpp  <- Generates foo.exe
   │  ├ bar.cpp  <- Generates bar.exe
   │  └ baz.cpp  <- Generates baz.exe
   │
   └─ src/       <- #include and link into `apps/` code
      └ <backend code...>
   

   2. The "designated" layout: apps/ should be treated as a source directory like src/ that implicitly uses the merged-header layout (headers and sources are sibling files). Any source files with a .main.* suffix are application entrypoint function source files, and each will produce an executable. All other source files are entrypoint source files that do not contain an entrypoint function, but otherwise provide support code for the application entrypoints. This is for projects with multiple entrypoints that have additional support code that are not themselves entrypoints, and which support code is not itself part of the "business logic" of the program.

   <root-or-submodule>
   │
   ├─ apps/               <- #include for everything in `apps/`
   │  ├ foo.main.cpp      <- Generates foo.exe
   │  └ foo_stuff/
   │    ├ cli_parser.hpp  <- Entrypoint code. Linked into `foo.exe`
   │    └ cli_parser.cpp  <- ''
   │
   └─ src/                <- #include and link into all `apps/` code
      └ <backend code...>
   

   3. The "submodule" layout: apps/ is treated as a submodule root. Each submodule of apps/ should contain at most one application entrypoint function source file. Each submodule that contains such a source file should produce an executable. Every submodule of apps/ should #include and link to the library defined by src/. Inter-linking and inclusion between submodules of apps/ is permitted. This is for large and complex applications that have a large amount of entrypoint code and multiple entrypoints, and is not recommended for smaller projects that can make do with designated or simple layouts.

   <root-or-submodule>
   │
   ├─ apps/
   │  ├─ foo/               <- Generates foo.exe
   │  │  └─ src/
   │  │     └ foo.main.cpp
   │  │
   │  └─ bar/               <- Generates bar.exe
   │     └─ src/
   │        ├ bar.main.hpp
   │        └ stuff.cpp     <- Support code for bar.exe
   │
   └─ src/                  <- #include and link into all `apps/` code
      └ <backend code...>
   

---

While I general prefer to keep number of tweakable options to a minimum, there are a few places (like here) where I think offering a choice is necessary. Not all projects are huge compiler infrastructure hybrid application/library suites like LLVM, and not all projects are small little utilities like echo. Both classes need to be supported with such a layout, with provisions for both the small and the enormous.

I know it's been a while, but "I'm not dead yet!"

Feedback appreciated!