32 bits time_t and Y2038 issue

I do not see substantial difference in build time between a single Makefile approach and recursive make with job server. That is on Linux and with optimizing compilers. Slower filesystem handling or ultra-fast compilers could make a difference.

Also, I am trying to be explicit in my Makefile-s. Normal 'make' rules check (search) for various insane possibilities, being explict limits need for searching.

Recursive make makes a lot of sense if build must be split into stages and when directory structure reflects dependences.

That's true.

But compared to have a wildcard search to include all .c and .cpp files in the source directories, maintaining file lists is still more than nothing!

However, the real benefit from using automatic file searches like this is two-fold. One is that you can't get it wrong - you can't forget to add the new file to the list, or remove deleted or renamed files from the list. The other - bigger - effect is that there is never any doubt about the files in the project. A file is in the project and build if and only if it is in one of the source directories. That consistency is very important to me - and to anyone else trying to look at the project. So any technical help in enforcing that is a good thing in my book.

I do sometimes have explicit lists for /directories/ - but not for files. I often have one branch in the source directory for my own code, and one branch for things like vendor SDKs and third-party code. I can then use stricter static warnings for my own code, without triggering lots of warnings in external code.

A simple rule of "all files are in the project" is more amenable to audit.

OK, that is not the classic recursive make pattern (ie. run make in each subdirectory). I do that (ie. building for multiple boards) using build scripts that are external to make.

cu Michael

Of course not --- the "route" in question was already "using CMake or Meson" after all.

The makers of Ninja are quite clear about that in their mission statement. Ninja scripts are strictly intended to only ever be written by machines, not humans, and they're intended to be executed _fast_. No creature comforts, no powerful flexibility; just raw speed.n,mn,m n

Possibly the single biggest eye-opener might be when you run a build twice in a row without changing anything else in between. So for the second run the entire job boils down to finding out that there's nothing left to do. Ninja is so much faster in that scenario that it's almost painful to watch Make, once you've seen Ninja do it.

Yes, David, thank you very much all of you for a very interesting discussion.

Recetly I included lwip in one of my project. I added it as a git submodule in a subfolder of my repository.

I don't know if you already played with lwip. This repo has several files and directories that aren't strictly needed during build process.

For example, there is test directory that shouldn't be included in the building process of the output binary. So, if you use recursive wildcard (lwip/*), at least you need a facility to exclude a subdirectory (lwip/test).

However this doesn't happen only to subfolders. Consider the folder lwip/src/apps/http. Here you have a simple http client and server implementantion (httpd.c and http_client.c). If you need only the client, you have to exclude httpd.c file from your build.

In this scenario writing explicitly the list of files to compile in the makefile seems a better alternative to wildcard.

Another possibility is to rename or delete the files/folders you don't need under lwip folder. However this means you will not have a clean copy of the lwip official repository.

This argument blindly assumes files matching the wildcard patterns must self-evidently be "old", and "still" in there. That assumption can be _wildly_ wrong. People will sometimes make backup copies of source files in situ, e.g. for experimentation. Source files can also get accidentally lost or moved.

Adding a source to the build on the sole justification that it exists, in a given place, IMHO undermines any remotely sane level of configuration management. Skipping a file simply because it's been lost is even worse.

Hunting for what source file the undefined reference in the final link was supposed to have come from, but didn't, is rather less fun than a clear message from Make that it cannot build foo.o because its source is nowhere to be found. The opposite job of hunting down duplicate definitions introduced by spare source files might be easier --- but then again, it might not be. Do you _always_ know, off the top of your head, whether the definition of function "get_bar" was supposed to be in dir1/dir2/baz.cpp or dir3/dir4/dir5/baz.cpp?

Which IMHO actually is the best argument _not_ to do it every time you run the build. And that includes not having make to do it for you, every time. All that wildcard discovery adds work to every build while introducing unnecessary risk to the build's reproducibility.

Setting up file lists using wildcards is a type of job best done just once, so after you've verified and fine-tuned the result, you save it and only repeat the procedure on massive additions or structural changes.

Keeping that list updated will also be less of a chore than enforcing a "thou shalt not put files in that folder lest they will be added to the build without your consent" policy.

Agreed - it is not the pattern that the famous paper warned against. But it /is/ recursive make. And in general, I think recursive make can potentially be useful in various ways, but you have to be very careful about how you use it in order to do so safely (and efficiently - but of course safety and correctness is the priority).

Most assumptions can be wildly wrong at times :-)

Yes, people do all sorts of things - I was merely describing what I think is the best way to organise files, in most circumstances.

Sometimes you do want to make a backup copies of a source file before doing some odd changes, and you don't want to do it via your source code system (git, subversion, whatever). This should be rare enough, and temporary enough, that you can usually put the copy in a completely separate directory without risking confusion. Or you simply make your copy of "file.c" as "file.c.old", "file.c.1", or similar - anything except "Copy of file.c".

And if you are losing or moving you files accidentally, you have bigger problems that can be solved by manual file lists!

If I have a project, the files in the project are in the project directory. Where else would they be? And what other files would I have in the project directory than project files? It makes no sense to me to have a random bunch of old, broken, lost or misplaced source files inside the source file directories of a project. If I remove a file from a project, I remove it from the project - and if I want to see the old file some time in the future, I can see it in the source code revision system.

As regards "lost" files, I don't know where to being to follow your argument. How do you "lose" files? And having lost a file, how could your build do anything other than skip it? Of course your build is likely to fail, at least during linking - regardless of whether you have manually-maintained file lists or automatic lists from wildcards.

How often have you "lost" a file accidentally? At the risk of making an assumption, I bet it is a much rarer event than including a new file in a project and forgetting to add it to your manual makefile lists, and having to figure out what went wrong in /that/ build.

Or for even more fun, you refactor to move a function from an existing C file that has grown too big, and put it into a new file and expand the function. But you've accidentally left the function in the old file, so your build works without complaint and yet no matter how many "printf" debug lines you add to the function, they never turn up in your testing

- because your build is still using the old file and old function, and does not catch the duplication at link time.

The /real/ problem with a haphazard source file organisation is when someone else has to look at the project (perhaps that person being your future self in ten years time). The new maintainer's task is to find all uses of the function "foo" and re-code them to use "bar" instead. They are faced with the fun of trying to figure out which of the 20 files that grep said used "foo" are actually relevant - which are part of the real build, which are part of experimental or alternative builds done sometimes, and which are junk leftover because a previous developer did no housekeeping.

Of course there are pros and cons to every way of organising files. And sometimes you need a variation of a standard rule - you need /some/ flexibility to adapt to the needs of any given project. But start with the rational setup of having files in the project source directory if and only if they are part of the project source. Then you can have the convenience and reliability of wildcard rules in your build setup. And /then/ you can add in exceptions if you have good reason for it - specify exceptions to the patterns, don't throw out the whole pattern.

Generally, yes, I do - at least for my own code or code that I am working with heavily. Amongst other things, I would avoid having two "baz.cpp" files. (Sometimes files from different upstream sources might share a name by coincidence, so the build has to tackle that safely, but humans get these mixed up more easily than computers.) And I have the directories and subdirectories named sensibly with files grouped appropriately, to make it easier to navigate.

But one thing I can be sure of with my system - "get_bar" is only defined in one place in one file. With a less regular approach with manual file lists and old (or moved, or "lost") files hanging around, maybe "get_bar" is defined in /both/ versions of "baz.cpp" and it is far from clear which one is actually used in the project.

There's no risk to build reproducibility. Why would there be? If you have the same files, you have the same wild-card generated lists. If you have different files, you have different generated lists - but you don't expect the same build binary from two different sets of source files!

It's the kind of job best done automatically, in a fraction of a millisecond, without risk of errors or getting out of date.

I also insist on a policy of not letting my cat walk across my keyboard while the editor is in focus. I don't consider that any more of a "chore" than a policy of putting the right files in the right place!

If the project can be compiled for different targets, you may have files that are used only for one target - stuff like i2c_stm32f0.c and i2c_stm32f1.c.

Both are project files, but only one is supposed to end up in the compilation. You may work around this by putting files in separate directories, but at some point you end up with lots of directories with only

This gets to the point of build configuration - make needs to know which files belong to a build configuration. Putting "#ifdef TARGET_STM32F0" around the whole C file is not a good way to do this in a larger project (not only because newer compilers complain that "ISO C forbids an empty translation unit").

Some optional features influence both make and the compile progress - at work, we decided to put that knowledge outside make, and generate sets of matching include files for make/c/c++ during the configure stage.

As you said, there are pros and cons - use what works for your project.

cu Michael

Depends on your workflow. I frequently do developement outside of source tree, then one can forget to copy file to the source tree. Explicit list means that you get clear build error when file is missing, that needs fixing anyway. Possibly less clear error when you add file without updating file list, but since normally adding a file is followed by make it is easy to find the reason.

A file is in the project if and only if it is in the source repository. Concerning "build", project normally allows optional/variant files and file is build if and only if it is needed in choosen configuration. Clearly, file not needed in any configuration has no place in source repository.

During developement in my work tree (different from source repository!) there may be some auxiliary file (it is rather infrequent and not a big deal anyway, I just mention how I work).

Well, for me (as mentioned above) "files in the project" and "build files" are different things.

Maybe your wildcard use is very simple, but year ago wildcards were important part in obfuscationg presence of maliciuous code in lzma.

But more important part is keeping info together, inside Makefile.

That is a possibility, yes - and I have had such cases and dealt with them by including or excluding specific directories. You are very unlikely to have /lots/ of directories with only one file, because your one project does not usually need lots of builds. And you also often have multiple device-specific files, not just one - especially for significantly different devices.

It is also not uncommon to see headers done like this :

// device.h

#if DEVICE == DEVICE_STM32F0 #include "device_stm32f0.h" #elif DEVICE == DEVICE_STM32F1 #include "device_stm32f1.h" #endif

That's less common for C or C++ files than for headers.

Add :

enum { this_file_may_be_intentionally_blank };

:-)

Indeed. But I find wildcard identification of files for the build to be many more pros, and fewer cons, than explicit lists - at least as the normal pattern.

My wildcards are often recursive, and cover different kinds of files, so they are not entirely simple. (That also makes it easier to re-use virtually the same makefiles in different projects.)

I admit I have been thinking primarily about work projects where commit access is only from a few people. If there are malicious actors involved, then probably any way to organise files and projects can be abused.

Agreed. That is a lot more important than whether the list of files in the build is generated from a wildcard pattern specified in the makefile, or from a manual list of files in the makefile.