Rebuilding targets after project configuration updates a config file made with configure_file()

[mitchburnett]

I have a project using c and cuda where I create a prj_config.h file to populate my project build parameters and create a static library. I generate the prj_config.h file using configuration_file() as part of my meson.build pulling the parameters as options that are set in the configuration data from meson_options.txt . The prj_config.h file is applied to the project as a c/cuda project argument via a -include prj_config.h flag.

Running meson setup -DNPARAM=value builddir and then meson compile in builddir the parameter is applied properly in creating the static library. If I reconfigure the option as meson --reconfigure -DNPARAM=newvalue and run a new meson compile the parameter reconfiguration rebuilds c targets but the cuda static library is not rebuilt. A meson compile --clean and meson compile will regenerate and the parameter is now applied with the new configuration.

Is this expected behavior when using cuda in meson? If I try the same thing with only c sources a meson --reconfigure -DNPARAM=newvalue and meson compile updates and is properly used in the static library. My meson version is 0.62.2.

The following is a very minimal example showing the described behavior. My meson.build and meson_options.txt,

project('prj', 'c', 'cuda')

nparam = get_option('NPARAM')

cdata = configuration_data()
cdata.set('NPARAM', nparam)
configure_file(output:'prj_config.h', configuration: cdata)

srcs = files('main.c')
libfoo_srcs = ('foo.cu')

add_project_arguments('-include', 'prj_config.h', language: ['c', 'cuda'])
cuda_dep = dependency('cuda')

libfoo = static_library('foo', libfoo_srcs, dependencies: cuda_dep)
executable('main', srcs, link_with: libfoo, dependencies: cuda_dep)

option('NPARAM', type: 'integer', value: 8)

and 4 source files: main.c, foo.cu, foo.h, defs.h. The cuda file does not include any kernel code just for simplicity to show the behavior.

main.c

#include <stdio.h>

#include "defs.h"
#include "foo.h"

int main() {
  printf("NPARAM=%u\n", NPARAM);
  int a = foobar();
  printf("a=%d\n", a);
  return 0;
}

foo.cu

#include "defs.h"
#include "foo.h"

int foobar() {
  return NPARAM;
}

foo.h

#ifndef FOO_H
#define FOO_H

#ifdef __cplusplus
extern "C" {
#endif

int foobar();

#ifdef __cplusplus
}
#endif

#endif

def.h

#ifndef DEFS_H
#define DEFS_H

#ifndef NPARAM
#define NPARAM 16
#endif

#endif

If I instead change the line add_project_arguments('-include', 'prj_config.h', language: ['c', 'cuda']) in my meson.build to be

add_project_arguments('-include', 'prj_config.h', language: 'c')
add_project_arguments(['-DNPARAM=@0@'.format(nparam)], language: 'cuda')

where the parameter is instead defined as a macro with a -D flag a call to meson --reconfigure results in that a meson compile rebuilds the static library target.

[eli-schwartz]

From the discussions around originally adding cuda support to Meson: #1003 (comment)

nvcc doesn't implement good Makefile depfile generation. A fuller explanation is in the source, but basically NVCC insists on doing depfile generation and compilation in two steps, doesn't support -MMD, -MQ or -MF but does support the inferior -MT, and there seems to be no easy way to square this with Ninja's needs (e.g. not possible to run two command for one rule, and nvcc prints its dependencies lists to stdout and not to a file).

cuda's backend in Meson doesn't implement get_dependency_gen_args(), so no depfile is created. As a result, ninja doesn't know when cuda's headers are updated and the source file needs to be rebuilt.

[mitchburnett]

Thank you!

It looks like nvcc does now support -MD, -MMD, -MF in the preprocessor. But, there is still no support for -MQ, just -MT. With -MD -MT and -MF, would this be sufficient for now until (if it ever happens) nvcc implements support for -MQ? Or will this cause too many issues?

This also presumes that the only requirement is to implement get_dependency_gen_args(). I could understand that just implementing get_dependency_gen_args() is the first step in a long laundry list of possible other tasks that would need to also take place across meson to make sure build dependencies are properly accounted for.

Although, I can report that at a minimum if get_dependency_gen_args() is implemented in mesonbuild/compilers/cuda.py as

    def get_dependency_gen_args(self, outtarget: str, outfile: str) -> T.List[str]:
        return ['-MD', '-MT', outtarget, '-MF', outfile]

and rerun this example, a --reconfigure -DNPARAM=newvalue does result in depfile generation of the config header for the target and it is rebuilt.

I am also able to run ./run_project_tests.py --only cuda with only the 15 sanitizer test failing. But, this test case seems to fail on a clean branch too.

[eli-schwartz]

The depfile handling is generic enough that implementing the required function for a compiler is all you need, which is why it works.

I'd be interested to hear @obilaniu weigh in on the topic. -MT is bad, but maybe better than nothing. For common projects the distinction shouldn't matter... one snag is if the source root is in a path with spaces. We automatically test that in CI, so a pull request implementing -MT for cuda will show any breakage that might exist.

[obilaniu]

@eli-schwartz @mitchburnett Yes, I was the author of that comment and a contributor to the current NVCC handling.

The CUDA dependency-generation support is a mess. There have been slight improvements over the generations of CUDA Toolkits in NVCC's command-line interface, but -MQ is still not supported. In newer NVCCs there exist options that could potentially enable a bypass, but it it would mean more and more NVCC version-dependent logic to exploit it opportunistically. It's overwhelming.

Yes, spaces in the paths will break the generated Makefiles, and there is no good solution for it. For a time I was working on a Python wrapper around nvcc that would --dryrun --verbose at first, but it was too difficult to get working. Even --dryrun --verbose fails to quote properly the commands it would have run when dumping, though the tool internally does maintain correct string separation.

---

To twist the knife in the wound, the NVCC compiler makes heavy use of temporary directories and files under /tmp for its "compilation trajectory", and one of the things it does is that it runs a tool several times (once per compute_XX microarchitecture) that disentangles the pure C++ from the pure CUDA code. It also generates stub files and #includes these stub files in other temporarily-generated source (!!!). Consequently, the host compiler (be it GCC, Clang or whatever) will generate object files that incorporate in themselves (either through __FILE__ or in debug info) transient, non-deterministic paths that incorporate, indirectly, the PID of the generating nvcc process. So NVCC output is non-reproducible by default.

If one uses nvcc --keep --keep-dir DIR/ with a stable path DIR/, then this second problem is avoided. If this DIR/ path further excludes spaces, and furthermore all paths into the source tree of relevance to NVCC exclude spaces, then in theory -MT would suffice.

---

I will give my Python NVCC wrapper attempt another kick, but I can't make promises. It's very hard to work around lack of proper nvcc -MQ.

[obilaniu]

Ah yes: And the reason I can't recommend simply using -MT is that the only way to know if -MT is sufficient is to

1. Parse the CUDA/C/C++ source code
2. With something of at least equal power and a similar self-identification as the GNU compiler's preprocessor, modified with CUDA's own defines
3. For all compute_XX "virtual architectures"
4. And output properly-quoted include paths when needed.
   ... which is exactly the function of -MQ.

If -MT works, it works - but you can't be sure of it. If -MT doesn't work, at best it produces garbage dependencies that can never be built (because no build rule exists for the hallucinated targets). At worst it's not even a valid Makefile snippet and you only know it after compilation and you can't fix it anyhow for lack of... -MQ.

[mitchburnett]

Thanks @obilaniu for taking a look here! This is really great information, and helps me get more familiar with the limitations really are here. Admittedly, much of this is learn (or relearn) as I go.

But, Yikes... this really does have some surprising non-trivial gotchas.

I have not used spaces or other characters that would need escaping in paths. That would make sense that depfile generation could break in that case. In addition, would it be possible that I get the appearance of it working when compiling a larger project because I only ever target a specific compute architecture (e.g., -arch=sm_86)? Although, in the simple example above I do not give an -arch=. But, I would understand and admit that there is no kernel code here. Possibly meaning that nvcc in its pass(es) just behaves as expected and something from the host compiler is leftover and I get lucky? Or is it possible that Ninja might be doing something?

Regardless, these are all things I could also be look into to see what is going on. Because something "is working" but considering what you have provided it is unclear the extent to which it does work. The most helpful thing then might be to describe that.

May I ask though, without this functionality, what is your "meson workflow" then? Are config files used? If so, then I am left to understand that a wipe of the build directory is needed to setup the build again? Or, am I missing something? If config files are not used, are macro's the preferred way to configure then?