NVAPI libnvidia-api.so.1 relevance

[jp7677]

The NVIDIA Linux driver ships libnvidia-api.so.1 since driver version 525. This library seems to be the Linux counterpart of nvapi64.dll from Windows. Lets use this issue to discuss ideas and its relevance to this project.

The version from the 525 driver contains the following methods (based on the function pointers described in the 520 NVAPI Open Source SDK) (thanks @Saancreed )

NvAPI_Initialize
NvAPI_Unload
NvAPI_GetErrorMessage
NvAPI_EnumPhysicalGPUs
NvAPI_GPU_GetShaderSubPipeCount
NvAPI_GPU_GetGpuCoreCount
NvAPI_GPU_GetFullName
NvAPI_GPU_GetPCIIdentifiers
NvAPI_GPU_GetBusType
NvAPI_GPU_GetBusId
NvAPI_GPU_GetVbiosRevision
NvAPI_GPU_GetVbiosOEMRevision
NvAPI_GPU_GetVbiosVersionString
NvAPI_GPU_GetCurrentPCIEDownstreamWidth
NvAPI_GPU_GetPhysicalFrameBufferSize
NvAPI_GPU_GetVirtualFrameBufferSize
NvAPI_GPU_GetBoardInfo
NvAPI_GPU_GetArchInfo
NvAPI_GPU_GetHDCPSupportStatus
NvAPI_GPU_GetTachReading
NvAPI_GPU_GetECCStatusInfo
NvAPI_GPU_GetECCConfigurationInfo
NvAPI_GPU_GetVirtualizationInfo
NvAPI_GPU_GetLicensableFeatures
NvAPI_GPU_GetVRReadyData
NvAPI_GPU_GetPstatesInfoEx
NvAPI_GPU_GetPstates20
NvAPI_GPU_GetCurrentPstate
NvAPI_GPU_GetDynamicPstatesInfoEx
NvAPI_GPU_GetThermalSettings
NvAPI_GPU_GetAllClockFrequencies
NvAPI_GPU_QueryIlluminationSupport
NvAPI_GPU_GetIllumination
NvAPI_GPU_SetIllumination
NvAPI_GPU_ClientIllumDevicesGetInfo
NvAPI_GPU_ClientIllumDevicesGetControl
NvAPI_GPU_ClientIllumDevicesSetControl
NvAPI_GPU_ClientIllumZonesGetInfo
NvAPI_GPU_ClientIllumZonesGetControl
NvAPI_GPU_ClientIllumZonesSetControl

[Saancreed]

Small correction: the library file is called libnvidia-api.so.1 and it was introduced in 525 series.

We only know about functions that are included in https://download.nvidia.com/XFree86/nvapi-open-source-sdk/R520-OpenSource.tar SDK which hasn't been updated to 525 release yet.

The program I used to check which functions are available is this:

#include "./nvapi.h"
#include "./nvapi_interface.h"

#include <dlfcn.h>
#include <iostream>
#include <memory>

int main()
{
    auto nvapi = std::unique_ptr<void, decltype(&dlclose)>{dlopen("libnvidia-api.so.1", RTLD_NOW), dlclose};
    auto nvapi_QueryInterface = reinterpret_cast<void *(*)(NvU32)>(dlsym(nvapi.get(), "nvapi_QueryInterface"));

    for (auto &i : nvapi_interface_table)
    {
        if (nvapi_QueryInterface(i.id))
            std::cout << "    Found: " << i.func << std::endl;
        else
            std::cerr << "NOT Found: " << i.func << std::endl;
    }
}

Might be useful to check again once NVIDIA releases R525 NVAPI SDK (build with -ldl).

[jp7677]

Small correction: the library file is called libnvidia-api.so.1 and it was introduced in 525 series.

Ah, thanks, corrected, that was a genuine typo ;).

[Etaash-mathamsetty]

I am going to check which functions are exported on windows out of curiosity, maybe linux only has a small number of them
edit: looking at nvapi.h it's already clear linux has only a small number (that we know of right now, they haven't released the R525 sdk yet)

[Saancreed]

Nvidia released R525 SDK sometime earlier this week and the only new function that can be found in libnvidia-api.so is NvAPI_GPU_GetRamBusWidth. There's also a bunch of new functions related to HDR but they seem to be Windows-only, at least in the latest driver release.

[jp7677]

Thanks for the heads up!
Hopefully we can merge Liam's PR soon. After that I'll also update to the new headers and then it is time for a new release.

[jp7677]

Regarding libnvidia-api.so, in its current state it could serve as a perfect nvml replacement because, well obviously, all those GPU_ related methods match exactly and no translation is needed like we have to do for nvml.

That said, we cannot unfortunately throw out the existing DXVK/Vulkan based sysinfo things and replace it with enumerating physical GPU's using libnvidia-api.so. The major showstopper (next to a few others, like honoring DXVK_DEVICE_FILTER, missing display related methods) is NvAPI_GPU_GetAdapterIdFromPhysicalGpu which returns the adapter LUID (needed for DLSS). libnvidia-api.so doesn't has this method, but even if i had an implementation, the LUID is generated by wine/wine-vulkan and thus not known to libnvidia-api.so. Doing the LUID magic in a wine-nvidia-api project would imho essentially just mirror the current sysinfo mechanics.

[Saancreed]

Another thing to look out for is how to properly correlate DXGI adapters / VkPhysicalDevices returned by DXVK and physical GPU handles returned by native libnvidia-api.so. NVML has the nice property of being able to find devices by PCI bus ID for which Vulkan gives us a tuple of (Domain, Bus, Device, Function) we can easily use. It can even find them by UUIDs if we were inclined to go that path. But with native NvPhysicalGpuHandles we are limited to what NVAPI can tell us about each handle it returns, which appears to be… not much.

Assuming that we have a PC with, let's say, four RTX 4090 GPUs (and a private power plant for them, of course), each one with the same full name, vendorId and deviceId but with distinct UUIDs and PCI bus IDs, how do we associate a VkPhysicalDevice with correct NvPhysicalGpuHandle?

(We'd also lose NvAPI_GPU_GetIrq if we were to dump NVML completely but that's not a big loss.)

[jp7677]

Yeah, that's indeed a blocking issue for now. I had hoped we can correlate by NvAPI_GPU_GetBusId and NvAPI_GPU_GetBusSlotId, which is the same as PCI bus ID/device ID. But unfortunately NvAPI_GPU_GetBusSlotId is currently not implemented by libnvidia-api.

[jp7677]

Out of curiosity I've looked again at this with NVIDIA 565. TLDR: Not much has changed.

The list of methods in libnvidia-api.so.1 which are also available in the known interface table are:

Available functions

NvAPI_Initialize
NvAPI_Unload
NvAPI_GetErrorMessage
NvAPI_EnumPhysicalGPUs
NvAPI_GPU_GetShaderSubPipeCount
NvAPI_GPU_GetGpuCoreCount
NvAPI_GPU_GetFullName
NvAPI_GPU_GetPCIIdentifiers
NvAPI_GPU_GetBusType
NvAPI_GPU_GetBusId
NvAPI_GPU_GetVbiosRevision
NvAPI_GPU_GetVbiosOEMRevision
NvAPI_GPU_GetVbiosVersionString
NvAPI_GPU_GetCurrentPCIEDownstreamWidth
NvAPI_GPU_GetPhysicalFrameBufferSize
NvAPI_GPU_GetVirtualFrameBufferSize
NvAPI_GPU_GetBoardInfo
NvAPI_GPU_GetRamBusWidth
NvAPI_GPU_GetArchInfo
NvAPI_GPU_GetHDCPSupportStatus
NvAPI_GPU_GetTachReading
NvAPI_GPU_GetECCStatusInfo
NvAPI_GPU_GetECCConfigurationInfo
NvAPI_GPU_GetVirtualizationInfo
NvAPI_GPU_GetLicensableFeatures
NvAPI_GPU_NVLINK_GetCaps
NvAPI_GPU_NVLINK_GetStatus
NvAPI_GPU_GetGPUInfo
NvAPI_GPU_GetVRReadyData
NvAPI_GPU_GetGspFeatures
NvAPI_GPU_GetPstatesInfoEx
NvAPI_GPU_GetPstates20
NvAPI_GPU_GetCurrentPstate
NvAPI_GPU_GetDynamicPstatesInfoEx
NvAPI_GPU_GetThermalSettings
NvAPI_GPU_GetAllClockFrequencies
NvAPI_GPU_QueryIlluminationSupport
NvAPI_GPU_GetIllumination
NvAPI_GPU_SetIllumination
NvAPI_GPU_ClientIllumDevicesGetInfo
NvAPI_GPU_ClientIllumDevicesGetControl
NvAPI_GPU_ClientIllumDevicesSetControl
NvAPI_GPU_ClientIllumZonesGetInfo
NvAPI_GPU_ClientIllumZonesGetControl
NvAPI_GPU_ClientIllumZonesSetControl
NvAPI_SYS_GetPhysicalGPUs

That said, not all of those are supported/usable on my setup (tested only get/query functions), which also probably makes sense:

Supported functions

nvapi_Initialize(): NVAPI_OK
nvapi_SYS_GetPhysicalGPUs(&gpus): NVAPI_NO_IMPLEMENTATION
nvapi_EnumPhysicalGPUs(&handle, &count): NVAPI_OK
nvapi_GPU_GetShaderSubPipeCount(handle, &shader_sub_pipe_count): NVAPI_OK
nvapi_GPU_GetGpuCoreCount(handle, &gpu_core_count): NVAPI_OK
nvapi_GPU_GetFullName(handle, full_name): NVAPI_OK
nvapi_GPU_GetPCIIdentifiers(handle, &device_id, &sub_system_id, &revision_id, &ext_device_id): NVAPI_OK
nvapi_GPU_GetBusType(handle, &bus_type): NVAPI_OK
nvapi_GPU_GetBusId(handle, &bus_id): NVAPI_OK
nvapi_GPU_GetVbiosRevision(handle, &vbios_revision): NVAPI_OK
nvapi_GPU_GetVbiosOEMRevision(handle, &vbios_oem_revision): NVAPI_OK
nvapi_GPU_GetVbiosVersionString(handle, vbios_version_string): NVAPI_OK
nvapi_GPU_GetCurrentPCIEDownstreamWidth(handle, &current_pcie_downstream_width): NVAPI_OK
nvapi_GPU_GetPhysicalFrameBufferSize(handle, &physical_frame_buffer_size): NVAPI_OK
nvapi_GPU_GetVirtualFrameBufferSize(handle, &virtual_frame_buffer_size): NVAPI_OK
nvapi_GPU_GetBoardInfo(handle, &board_info): NVAPI_NOT_SUPPORTED
nvapi_GPU_GetRamBusWidth(handle, &ram_bus_width): NVAPI_OK
nvapi_GPU_GetArchInfo(handle, &arch_info): NVAPI_OK
nvapi_GPU_GetHDCPSupportStatus(handle, &hdcp_support_status): NVAPI_OK
nvapi_GPU_GetTachReading(handle, &tach_reading): NVAPI_NOT_SUPPORTED
nvapi_GPU_GetECCStatusInfo(handle, &ecc_status_info): NVAPI_OK
nvapi_GPU_GetECCConfigurationInfo(handle, &ecc_configuration_info): NVAPI_NOT_SUPPORTED
nvapi_GPU_GetVirtualizationInfo(handle, &virtualization_info): NVAPI_OK
nvapi_GPU_GetLicensableFeatures(handle, &licensable_features): NVAPI_ERROR
nvapi_GPU_NVLINK_GetCaps(handle, &nvlink_get_caps): NVAPI_OK
nvapi_GPU_NVLINK_GetStatus(handle, &nvlink_get_status): NVAPI_OK
nvapi_GPU_GetGPUInfo(handle, &gpu_info): NVAPI_OK
nvapi_GPU_GetVRReadyData(handle, &gpu_vr_ready): NVAPI_ERROR
nvapi_GPU_GetGspFeatures(handle, &gsp_info): NVAPI_OK
nvapi_GPU_GetPstatesInfoEx(handle, &perf_pstates_info, input_Flags): NVAPI_OK
nvapi_GPU_GetPstates20(handle, &perf_pstates20_info): NVAPI_OK
nvapi_GPU_GetCurrentPstate(handle, &perf_pstate_id): NVAPI_OK
nvapi_GPU_GetDynamicPstatesInfoEx(handle, &dynamic_pstates_info): NVAPI_OK
nvapi_GPU_GetThermalSettings(handle, sensorIndex, &thermal_settings): NVAPI_OK
nvapi_GPU_GetAllClockFrequencies(handle, &clock_frequencies): NVAPI_OK
nvapi_GPU_QueryIlluminationSupport(&query_ilumination_support): NVAPI_OK
nvapi_GPU_GetIllumination(&get_ilumination_support): NVAPI_NOT_SUPPORTED
nvapi_GPU_ClientIllumDevicesGetInfo(handle, &client_illum_device_info_params): NVAPI_OK
nvapi_GPU_ClientIllumDevicesGetControl(handle, &client_illum_device_control_params): NVAPI_OK
nvapi_GPU_ClientIllumZonesGetInfo(handle, &client_illum_zone_info_params): NVAPI_OK
nvapi_GPU_ClientIllumZonesGetControl(handle, &client_illum_zone_control_params): NVAPI_OK
nvapi_Unload(): NVAPI_OK

Tested with:

meson.build

project(
    'test-libnvidia',
    ['cpp'],
    default_options: ['cpp_std=c++20'],
    meson_version: '>= 1.1',
)

nvapi_headers = include_directories('../external/nvapi')

executable('test-libnvidia', sources: ['test-libnvidia.cpp'], include_directories : [ nvapi_headers ])

test-libnvidia.cpp

#include <algorithm>
#include <memory>
#include <iostream>
#include <cstring>

#include <dlfcn.h>

#define __cdecl           // NOLINT(*-reserved-identifier)
#define __NVAPI_EMPTY_SAL // NOLINT(*-reserved-identifier)
#include "nvapi.h"
#include "nvapi_interface.h"
#undef __NVAPI_EMPTY_SAL
#undef __cdecl

int main() {
    const auto nvapi_module = std::unique_ptr<void, decltype([](void* h) { dlclose(h); })>{dlopen("libnvidia-api.so.1", RTLD_NOW)};
    const auto nvapi_QueryInterface = reinterpret_cast<void* (*)(NvU32)>(dlsym(nvapi_module.get(), "nvapi_QueryInterface"));

    std::cout << "Found NVAPI methods:" << std::endl;
    for (auto& [func, id] : nvapi_interface_table) {
        if (nvapi_QueryInterface(id))
            std::cout << func << std::endl;
    }

    auto nvapi_PFN = [](const char* name) {
        const auto it = std::ranges::find_if(
            nvapi_interface_table,
            [name](const auto& item) { return strcmp(item.func, name) == 0; });

        if (it == std::end(nvapi_interface_table))
            throw std::runtime_error("nvapi_PFN: unknown function");

        return it->id;
    };

#define GETNVAPIPROCADDR(x) auto nvapi_##x = reinterpret_cast<decltype(&NvAPI_##x)>(nvapi_QueryInterface(nvapi_PFN("NvAPI_" #x)));
#pragma GCC diagnostic ignored "-Wunused-variable"

    GETNVAPIPROCADDR(Initialize)
    GETNVAPIPROCADDR(Unload)
    GETNVAPIPROCADDR(GetErrorMessage)
    GETNVAPIPROCADDR(EnumPhysicalGPUs)
    GETNVAPIPROCADDR(GPU_GetShaderSubPipeCount)
    GETNVAPIPROCADDR(GPU_GetGpuCoreCount)
    GETNVAPIPROCADDR(GPU_GetFullName)
    GETNVAPIPROCADDR(GPU_GetPCIIdentifiers)
    GETNVAPIPROCADDR(GPU_GetBusType)
    GETNVAPIPROCADDR(GPU_GetBusId)
    GETNVAPIPROCADDR(GPU_GetVbiosRevision)
    GETNVAPIPROCADDR(GPU_GetVbiosOEMRevision)
    GETNVAPIPROCADDR(GPU_GetVbiosVersionString)
    GETNVAPIPROCADDR(GPU_GetCurrentPCIEDownstreamWidth)
    GETNVAPIPROCADDR(GPU_GetPhysicalFrameBufferSize)
    GETNVAPIPROCADDR(GPU_GetVirtualFrameBufferSize)
    GETNVAPIPROCADDR(GPU_GetBoardInfo)
    GETNVAPIPROCADDR(GPU_GetRamBusWidth)
    GETNVAPIPROCADDR(GPU_GetArchInfo)
    GETNVAPIPROCADDR(GPU_GetHDCPSupportStatus)
    GETNVAPIPROCADDR(GPU_GetTachReading)
    GETNVAPIPROCADDR(GPU_GetECCStatusInfo)
    GETNVAPIPROCADDR(GPU_GetECCConfigurationInfo)
    GETNVAPIPROCADDR(GPU_GetVirtualizationInfo)
    GETNVAPIPROCADDR(GPU_GetLicensableFeatures)
    GETNVAPIPROCADDR(GPU_NVLINK_GetCaps)
    GETNVAPIPROCADDR(GPU_NVLINK_GetStatus)
    GETNVAPIPROCADDR(GPU_GetGPUInfo)
    GETNVAPIPROCADDR(GPU_GetVRReadyData)
    GETNVAPIPROCADDR(GPU_GetGspFeatures)
    GETNVAPIPROCADDR(GPU_GetPstatesInfoEx)
    GETNVAPIPROCADDR(GPU_GetPstates20)
    GETNVAPIPROCADDR(GPU_GetCurrentPstate)
    GETNVAPIPROCADDR(GPU_GetDynamicPstatesInfoEx)
    GETNVAPIPROCADDR(GPU_GetThermalSettings)
    GETNVAPIPROCADDR(GPU_GetAllClockFrequencies)
    GETNVAPIPROCADDR(GPU_QueryIlluminationSupport)
    GETNVAPIPROCADDR(GPU_GetIllumination)
    GETNVAPIPROCADDR(GPU_SetIllumination)
    GETNVAPIPROCADDR(GPU_ClientIllumDevicesGetInfo)
    GETNVAPIPROCADDR(GPU_ClientIllumDevicesGetControl)
    GETNVAPIPROCADDR(GPU_ClientIllumDevicesSetControl)
    GETNVAPIPROCADDR(GPU_ClientIllumZonesGetInfo)
    GETNVAPIPROCADDR(GPU_ClientIllumZonesGetControl)
    GETNVAPIPROCADDR(GPU_ClientIllumZonesSetControl)
    GETNVAPIPROCADDR(SYS_GetPhysicalGPUs)

#pragma GCC diagnostic pop
#undef GETNVAPIPROCADDR

    NvAPI_Status status{};
    auto print_result = [nvapi_GetErrorMessage](const char* call, const NvAPI_Status nvapi_status) {
        NvAPI_ShortString status_message{};
        nvapi_GetErrorMessage(nvapi_status, status_message);
        std::cout << call << ": " << status_message << std::endl;
    };

#define CALL_NVAPI(x) \
    status = x;       \
    print_result(#x, status);

    std::cout << std::endl;
    std::cout << "Calling NVAPI:" << std::endl;

    CALL_NVAPI(nvapi_Initialize())

    NV_PHYSICAL_GPUS gpus{};
    gpus.version = NV_PHYSICAL_GPUS_VER;
    CALL_NVAPI(nvapi_SYS_GetPhysicalGPUs(&gpus))

    NvPhysicalGpuHandle handle{};
    NvU32 count{};
    CALL_NVAPI(nvapi_EnumPhysicalGPUs(&handle, &count))

    if (count != 1)
        throw std::runtime_error("Cannot deal with no or more than one GPU");

    NvU32 shader_sub_pipe_count;
    CALL_NVAPI(nvapi_GPU_GetShaderSubPipeCount(handle, &shader_sub_pipe_count))

    NvU32 gpu_core_count;
    CALL_NVAPI(nvapi_GPU_GetGpuCoreCount(handle, &gpu_core_count))

    NvAPI_ShortString full_name{};
    CALL_NVAPI(nvapi_GPU_GetFullName(handle, full_name))

    NvU32 device_id{};
    NvU32 sub_system_id{};
    NvU32 revision_id{};
    NvU32 ext_device_id{};
    CALL_NVAPI(nvapi_GPU_GetPCIIdentifiers(handle, &device_id, &sub_system_id, &revision_id, &ext_device_id))

    NV_GPU_BUS_TYPE bus_type{};
    CALL_NVAPI(nvapi_GPU_GetBusType(handle, &bus_type))

    NvU32 bus_id{};
    CALL_NVAPI(nvapi_GPU_GetBusId(handle, &bus_id))

    NvU32 vbios_revision{};
    CALL_NVAPI(nvapi_GPU_GetVbiosRevision(handle, &vbios_revision))

    NvU32 vbios_oem_revision{};
    CALL_NVAPI(nvapi_GPU_GetVbiosOEMRevision(handle, &vbios_oem_revision))

    NvAPI_ShortString vbios_version_string{};
    CALL_NVAPI(nvapi_GPU_GetVbiosVersionString(handle, vbios_version_string))

    NvU32 current_pcie_downstream_width{};
    CALL_NVAPI(nvapi_GPU_GetCurrentPCIEDownstreamWidth(handle, &current_pcie_downstream_width))

    NvU32 physical_frame_buffer_size{};
    CALL_NVAPI(nvapi_GPU_GetPhysicalFrameBufferSize(handle, &physical_frame_buffer_size))

    NvU32 virtual_frame_buffer_size{};
    CALL_NVAPI(nvapi_GPU_GetVirtualFrameBufferSize(handle, &virtual_frame_buffer_size))

    NV_BOARD_INFO board_info{};
    board_info.version = NV_BOARD_INFO_VER;
    CALL_NVAPI(nvapi_GPU_GetBoardInfo(handle, &board_info))

    NvU32 ram_bus_width{};
    CALL_NVAPI(nvapi_GPU_GetRamBusWidth(handle, &ram_bus_width))

    NV_GPU_ARCH_INFO arch_info{};
    arch_info.version = NV_GPU_ARCH_INFO_VER;
    CALL_NVAPI(nvapi_GPU_GetArchInfo(handle, &arch_info))

    NV_GPU_GET_HDCP_SUPPORT_STATUS hdcp_support_status{};
    hdcp_support_status.version = NV_GPU_GET_HDCP_SUPPORT_STATUS_VER;
    CALL_NVAPI(nvapi_GPU_GetHDCPSupportStatus(handle, &hdcp_support_status))

    NvU32 tach_reading{};
    CALL_NVAPI(nvapi_GPU_GetTachReading(handle, &tach_reading))

    NV_GPU_ECC_STATUS_INFO ecc_status_info{};
    ecc_status_info.version = NV_GPU_ECC_STATUS_INFO_VER;
    CALL_NVAPI(nvapi_GPU_GetECCStatusInfo(handle, &ecc_status_info))

    NV_GPU_ECC_CONFIGURATION_INFO ecc_configuration_info{};
    ecc_configuration_info.version = NV_GPU_ECC_CONFIGURATION_INFO_VER;
    CALL_NVAPI(nvapi_GPU_GetECCConfigurationInfo(handle, &ecc_configuration_info))

    NV_GPU_VIRTUALIZATION_INFO virtualization_info{};
    virtualization_info.version = NV_GPU_VIRTUALIZATION_INFO_VER;
    CALL_NVAPI(nvapi_GPU_GetVirtualizationInfo(handle, &virtualization_info))

    NV_LICENSABLE_FEATURES licensable_features;
    licensable_features.version = NV_LICENSABLE_FEATURES_VER;
    CALL_NVAPI(nvapi_GPU_GetLicensableFeatures(handle, &licensable_features))

    NVLINK_GET_CAPS nvlink_get_caps{};
    nvlink_get_caps.version = NVLINK_GET_CAPS_VER;
    CALL_NVAPI(nvapi_GPU_NVLINK_GetCaps(handle, &nvlink_get_caps))

    NVLINK_GET_STATUS nvlink_get_status{};
    nvlink_get_status.version = NVLINK_GET_STATUS_VER;
    CALL_NVAPI(nvapi_GPU_NVLINK_GetStatus(handle, &nvlink_get_status))

    NV_GPU_INFO gpu_info{};
    gpu_info.version = NV_GPU_INFO_VER;
    CALL_NVAPI(nvapi_GPU_GetGPUInfo(handle, &gpu_info))

    NV_GPU_VR_READY gpu_vr_ready{};
    gpu_vr_ready.version = NV_GPU_VR_READY_VER;
    CALL_NVAPI(nvapi_GPU_GetVRReadyData(handle, &gpu_vr_ready))

    NV_GPU_GSP_INFO gsp_info{};
    gsp_info.version = NV_GPU_GSP_INFO_VER;
    CALL_NVAPI(nvapi_GPU_GetGspFeatures(handle, &gsp_info))

    NV_GPU_PERF_PSTATES_INFO perf_pstates_info{};
    perf_pstates_info.version = NV_GPU_PERF_PSTATES_INFO_VER;
    NvU32 input_Flags{};
    CALL_NVAPI(nvapi_GPU_GetPstatesInfoEx(handle, &perf_pstates_info, input_Flags))

    NV_GPU_PERF_PSTATES20_INFO perf_pstates20_info{};
    perf_pstates20_info.version = NV_GPU_PERF_PSTATES20_INFO_VER;
    CALL_NVAPI(nvapi_GPU_GetPstates20(handle, &perf_pstates20_info))

    NV_GPU_PERF_PSTATE_ID perf_pstate_id{};
    CALL_NVAPI(nvapi_GPU_GetCurrentPstate(handle, &perf_pstate_id))

    NV_GPU_DYNAMIC_PSTATES_INFO_EX dynamic_pstates_info{};
    dynamic_pstates_info.version = NV_GPU_DYNAMIC_PSTATES_INFO_EX_VER;
    CALL_NVAPI(nvapi_GPU_GetDynamicPstatesInfoEx(handle, &dynamic_pstates_info))

    NvU32 sensorIndex{};
    NV_GPU_THERMAL_SETTINGS thermal_settings{};
    thermal_settings.version = NV_GPU_THERMAL_SETTINGS_VER;
    CALL_NVAPI(nvapi_GPU_GetThermalSettings(handle, sensorIndex, &thermal_settings))

    NV_GPU_CLOCK_FREQUENCIES clock_frequencies{};
    clock_frequencies.version = NV_GPU_CLOCK_FREQUENCIES_VER;
    CALL_NVAPI(nvapi_GPU_GetAllClockFrequencies(handle, &clock_frequencies))

    NV_GPU_QUERY_ILLUMINATION_SUPPORT_PARM query_ilumination_support{};
    query_ilumination_support.version = NV_GPU_QUERY_ILLUMINATION_SUPPORT_PARM_VER;
    query_ilumination_support.hPhysicalGpu = handle;
    CALL_NVAPI(nvapi_GPU_QueryIlluminationSupport(&query_ilumination_support))

    NV_GPU_GET_ILLUMINATION_PARM get_ilumination_support{};
    get_ilumination_support.version = NV_GPU_GET_ILLUMINATION_PARM_VER;
    get_ilumination_support.hPhysicalGpu = handle;
    CALL_NVAPI(nvapi_GPU_GetIllumination(&get_ilumination_support))

    NV_GPU_CLIENT_ILLUM_DEVICE_INFO_PARAMS client_illum_device_info_params{};
    client_illum_device_info_params.version = NV_GPU_CLIENT_ILLUM_DEVICE_INFO_PARAMS_VER;
    CALL_NVAPI(nvapi_GPU_ClientIllumDevicesGetInfo(handle, &client_illum_device_info_params))

    NV_GPU_CLIENT_ILLUM_DEVICE_CONTROL_PARAMS client_illum_device_control_params{};
    client_illum_device_control_params.version = NV_GPU_CLIENT_ILLUM_DEVICE_CONTROL_PARAMS_VER;
    CALL_NVAPI(nvapi_GPU_ClientIllumDevicesGetControl(handle, &client_illum_device_control_params))

    NV_GPU_CLIENT_ILLUM_ZONE_INFO_PARAMS client_illum_zone_info_params{};
    client_illum_zone_info_params.version = NV_GPU_CLIENT_ILLUM_ZONE_INFO_PARAMS_VER;
    CALL_NVAPI(nvapi_GPU_ClientIllumZonesGetInfo(handle, &client_illum_zone_info_params))

    NV_GPU_CLIENT_ILLUM_ZONE_CONTROL_PARAMS client_illum_zone_control_params{};
    client_illum_zone_control_params.version = NV_GPU_CLIENT_ILLUM_ZONE_CONTROL_PARAMS_VER;
    CALL_NVAPI(nvapi_GPU_ClientIllumZonesGetControl(handle, &client_illum_zone_control_params))

    CALL_NVAPI(nvapi_Unload())
}