What is the most idiomatic way to find Windows SDK tools?

[usernameak]

What is the most idiomatic way to find Windows SDK tools? Right now I am doing this:

import("core.tool.toolchain")
import("core.project.config")

import("lib.detect.find_program")
import("lib.detect.find_programver")

local function _find_windows_sdk_paths(opt)
    local msvc = toolchain.load("msvc")
    if not msvc or not msvc:check() then
        return
    end

    local vcvars = msvc:config("vcvars")
    if not vcvars or not vcvars.WindowsSdkDir or not vcvars.WindowsSDKVersion then
        return
    end
    
    local toolchain = opt.toolchain
    local arch = msvc:arch()
    local bindir = path.join(vcvars.WindowsSdkDir, "bin", vcvars.WindowsSDKVersion, arch)
    if os.isdir(bindir) then
        opt.paths = table.wrap(opt.paths)
        table.insert(opt.paths, bindir)
    end
end

-- find fxc
--
-- @param opt   the argument options, e.g. {version = true}
--
-- @return      program, version
--
-- @code
-- local fxc = find_fxc()
-- @endcode
--
function main(opt)
    opt         = opt or {}
    opt.parse   = opt.parse or function (output) return output:match("Microsoft %(R%) Direct3D Shader Compiler (%d+%.%d+)") end
    
    _find_windows_sdk_paths(opt)
    
    -- because apparently fxc /? returns status code of 1... should I just PR to xmake with option to change expected exit code?
    opt.check = opt.check or function(program)
        local outfile = os.tmpfile()
        local status_code, errors = os.execv(program, {"/?"}, {envs = opt.envs, stdout = outfile, stderr = outfile, shell = opt.shell, try = true})
        os.rm(outfile)
        
        if status_code ~= 1 and status_code ~= 0 then
            os.raise(errors)
        end
        
        return errors
    end

    local version = nil
    local program = find_program(opt.program or "fxc", opt)
    
    opt.command = opt.command or function()
        local outfile = os.tmpfile()
        
        local status_code, errors = os.execv(program, {"/?"}, {envs = opt.envs, stdout = outfile, stderr = outfile, shell = opt.shell, try = true})
        local outdata = io.readfile(outfile)
        os.rm(outfile)
        
        if status_code ~= 1 and status_code ~= 0 then
            os.raise(errors)
        end
        
        return outdata
    end
    
    if program and opt and opt.version then
        version = find_programver(program, opt)
    end
    
    return program, version
end

However, this seems very verbose (and is duplicating a lot of code from xmake's own codebase - this is derived from https://github.com/xmake-io/xmake/blob/master/xmake/modules/detect/tools/find_midl.lua code). Besides, xmake itself doesn't seem to have consistent behavior around this - find_midl and find_rc do it through env (which doesn't always seem to work), and find_dxc does something else. Whatever I wrote here seems to be the most reliable way regardless of the environment. Though DXC is a special case because you generally don't want the one from Windows SDK if another version from a different source exists, as the Windows SDK version is generally somewhat outdated. This does not, however, apply to FXC, as that tool only appears in Windows SDK currently (as well as the legacy DirectX SDK, but that one is deprecated).

Just directly calling find_program without preparing paths doesn't work very well -- it expects Windows SDK to be in the user's PATH (which it might not be if xmake is not invoked from VC developer command prompt).

What is the best way to do this?

also: private.utils.toolchain seems to have a lot of nice tools for toolchain detection, but it's not a public API unfortunately...

[waruqi]

However, this seems very verbose (and is duplicating a lot of code from xmake's own codebase - this is derived from https://github.com/xmake-io/xmake/blob/master/xmake/modules/detect/tools/find_midl.lua code).

This involves just a small snippet of code; why is your implementation so complex? opt.envs already includes all vs envs.

[usernameak]

Because the simpler approach found in find_midl does not work reliably on anything but the vsxmake generator -- it fails when invoking xmake build directly e.g. from a shell that doesn't have MSVC toolchain on PATH (but what I wrote works in that case, as it pulls from toolchain detection logic); it also doesn't work with the vstudio generator; can provide a full minimal reproducible example tomorrow if you want to check it out (though really you can just call the script with xmake l and compare it yourself with both msvc:config("vcvars") and opt.envs, while not having fxc.exe in your PATH).

opt.envs already includes all vs envs.

I am in fact not passing anything to it.

        import("modules.detect.tools.find_fxc")
    
        local fxc_program = find_fxc()
        
        if not fxc_program then
            os.raise("fxc not found")
        end
    

[waruqi]

I don't have a Windows environment, but opt.envs should contain vs envs if it's compiled for the Windows platform.

xmake f -c
xmake -rv

you can debug it.

xmake/xmake/toolchains/msvc/load.lua

Line 53 in ee1256b

toolchain_utils.add_vsenvs(toolchain)

xmake/xmake/core/tool/toolchain.lua

Line 587 in ee1256b

envs = self:get("runenvs")})

[usernameak]

I made a minimal reproducible example to make sure we're on the same page: https://github.com/usernameak/xmake-experiments . In neither case, opt.envs contains anything. Note that this is running under a standard command prompt, not VS Developer Command Prompt.

Logs from both variants:

find_fxc_test_simple_nonworking:

W:\src\xmake-experiments\find_fxc_test_simple_nonworking>xmake f -c
checking for platform ... windows (x64)
checking for Microsoft C/C++ Compiler (x64) ... ok
checking for Microsoft Visual Studio (x64) version ... 2022

W:\src\xmake-experiments\find_fxc_test_simple_nonworking>xmake -rv
opt.envs:
opt.paths:
opt.envs:
opt.paths:
checking for C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64\cl.exe ... c:\program files\microsoft visual studio\2022\community\vc\tools\msvc\14.44.35207\bin\hostx64\x64\cl.exe
program being checked fxc.exe
checking for fxc ... no
checking for flags (-std:c++20) ... no
checking for flags (-std:c++latest) ... ok
[ 15%]: <find_fxc_test> compiling.release src\test.cpp
"C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\bin\\HostX64\\x64\\cl.exe" -c -nologo -MD -std:c++latest -Ibuild\.gens\find_fxc_test\windows\x64\release\generated_shader_includes /EHsc -Fobuild\.objs\find_fxc_test\windows\x64\release\src\test.cpp.obj src\test.cpp
checking for flags (cl_sourceDependencies) ... ok
error: fxc not found

find_fxc_test_working:

W:\src\xmake-experiments\find_fxc_test_working>xmake f -c
checking for platform ... windows (x64)
checking for Microsoft C/C++ Compiler (x64) ... ok
checking for Microsoft Visual Studio (x64) version ... 2022

W:\src\xmake-experiments\find_fxc_test_working>xmake -rv
opt.envs:
opt.paths:  {
  "C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64"
}

program being checked C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64\fxc.exe
opt.envs:
opt.paths:  {
  "C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64"
}

checking for C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64\cl.exe ... c:\program files\microsoft visual studio\2022\community\vc\tools\msvc\14.44.35207\bin\hostx64\x64\cl.exe
checking for fxc ... C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64\fxc
[ 15%]: <find_fxc_test> generating.fxc: src\test.vp.hlsl(vs_4_0) -> build\.gens\find_fxc_test\windows\x64\release\generated_shader_includes\shaders\test_vp.h
C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64\fxc /nologo /Fh build\.gens\find_fxc_test\windows\x64\release\generated_shader_includes\shaders\test_vp.h /Vn g_spriteVertexShaderData /T vs_4_0 /E main /Zpc /Zi /O2 src\test.vp.hlsl
compilation header save succeeded; see W:\src\xmake-experiments\find_fxc_test_working\build\.gens\find_fxc_test\windows\x64\release\generated_shader_includes\shaders\test_vp.h
checking for flags (-std:c++20) ... ok
[ 31%]: <find_fxc_test> compiling.release src\test.cpp
"C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\bin\\HostX64\\x64\\cl.exe" -c -nologo -MD -std:c++20 -Ibuild\.gens\find_fxc_test\windows\x64\release\generated_shader_includes /EHsc -Fobuild\.objs\find_fxc_test\windows\x64\release\src\test.cpp.obj src\test.cpp
checking for flags (cl_sourceDependencies) ... ok
[ 31%]: <find_fxc_test> generating.fxc: src\test.fp.hlsl(ps_4_0) -> build\.gens\find_fxc_test\windows\x64\release\generated_shader_includes\shaders\test_fp.h
C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64\fxc /nologo /Fh build\.gens\find_fxc_test\windows\x64\release\generated_shader_includes\shaders\test_fp.h /Vn g_spritePixelShaderData /T ps_4_0 /E main /Zpc /Zi /O2 src\test.fp.hlsl
compilation header save succeeded; see W:\src\xmake-experiments\find_fxc_test_working\build\.gens\find_fxc_test\windows\x64\release\generated_shader_includes\shaders\test_fp.h
checking for link.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64\link.exe
checking for the static library archiver (ar) ... link.exe
[ 63%]: <find_fxc_test> archiving.release find_fxc_test.lib
"C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\bin\\HostX64\\x64\\link.exe" -lib -nologo -machine:x64 -out:build\windows\x64\release\find_fxc_test.lib build\.objs\find_fxc_test\windows\x64\release\src\test.cpp.obj
checking for cl.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64\cl.exe
checking for the c compiler (cc) ... cl.exe
[100%]: build ok, spent 0.985s