Skip to content

Unify catalog driver scrapers into a shared module + data tables#2

Merged
jm2 merged 1 commit into
mainfrom
unify-catalog-driver-scrapers
Jun 15, 2026
Merged

Unify catalog driver scrapers into a shared module + data tables#2
jm2 merged 1 commit into
mainfrom
unify-catalog-driver-scrapers

Conversation

@jm2

@jm2 jm2 commented Jun 15, 2026

Copy link
Copy Markdown
Owner

Summary

Unifies the six Microsoft Update Catalog driver scrapers into one catalogscrape/CatalogScrape.psm1 engine + per-vendor .psd1 device tables + thin Get-*Drivers.ps1 shims (still auto-discovered by build_win11pxe.ps1). ~120 lines of copy-pasted, drifted boilerplate collapse to one place.

Correctness fixes (live-verified against the catalog)

  • E810 silently acquired nothing — the "Windows 11" suffix zeroes a datacenter-NIC query. Now bare HWID (0 → 4 results).
  • Marvell AQC111U USB silently acquired nothing — VID_1D6A is the PCIe vendor id; the real USB id is VID_2ECA&PID_C101 (0 → 20 results, Aquantia v1.8.0.0). Dropped the dead TRENDnet/ASIX entries.
  • Selection is now highest-[version]-first (catalog date as tiebreak), fixing Realtek-USB (stale 2016 catalog date), Intel re-release, and MediaTek stale preferred-branch — all of which picked an older driver than the catalog offered.
  • Per-request timeouts, culture-invariant date parsing, tolerant version parsing, pnputil /subdirs, and broken-extract cleanup.

New coverage

  • Intel X520 (82599, bare HWID), Broadcom NetXtreme-E/C 10–100G, and FastLinQ 41xxx/45xxx folded into the Marvell module (FastLinQ is Marvell-owned).

25-row cap

  • Dual-query union (Expand-CsQuery): HWID queries are sampled from both the bare and + Windows 11 relevance windows, deduped, highest version wins (replaces the per-device AppendOsToken suffix guess). The catalog caps results at 25 relevance rows and rejects scripted sort/pagination postbacks (confirmed for both the MSCatalog module and a hand-rolled VIEWSTATE replay), so two windows is the durable, dependency-free option.

Validation

  • 68 network-free unit tests (catalogscrape/Test-CatalogScrape.ps1), lint clean, plus live end-to-end engine checks (Linux PowerShell 7) for every fix.
  • Windows-only steps (Authenticode / expand.exe / pnputil) are copied verbatim from the originals (can't run on Linux).

🤖 Generated with Claude Code

Replace six drifted copy-paste Get-*Drivers.ps1 scrapers with a shared
catalogscrape/CatalogScrape.psm1 engine, per-vendor .psd1 device tables,
and thin shims (still auto-discovered by build_win11pxe.ps1).

Correctness fixes (verified against the live Microsoft Update Catalog):
- E810 and Marvell AQC111U-USB returned 0 results and were silently
  skipped every run. E810: the "Windows 11" suffix zeroes a datacenter
  NIC query. AQC111U: VID_1D6A is the PCIe vendor id, not USB -> use the
  first-party VID_2ECA&PID_C101.
- Selection is now highest-[version]-first (catalog date as tiebreak),
  fixing Realtek-USB (stale catalog date), Intel re-release, and MediaTek
  stale preferred-branch, which all picked an older driver than offered.
- Add per-request timeouts, culture-invariant date parsing, tolerant
  version parsing, pnputil /subdirs, and broken-extract cleanup.

New coverage:
- Intel X520 (82599), Broadcom NetXtreme-E/C (10-100G), and QLogic
  FastLinQ (41xxx/45xxx) folded into the Marvell module (FastLinQ is
  Marvell-owned).

Durability (25-row cap):
- Dual-query union (Expand-CsQuery): HWID queries are sampled from both
  the bare and "+Windows 11" relevance windows, deduped, highest version
  wins. The catalog caps results at 25 relevance rows and rejects scripted
  sort/pagination postbacks, so this is the dependency-free way to beat
  the cap (replaces the per-device AppendOsToken suffix guess).

Adds catalogscrape/Test-CatalogScrape.ps1 (68 network-free unit tests).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jm2 jm2 merged commit 3d58f36 into main Jun 15, 2026
6 checks passed

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors several vendor-specific driver scraping scripts into thin shims that utilize a new, shared PowerShell module (CatalogScrape.psm1) driven by vendor-specific .psd1 configuration files, and introduces a unit test suite (Test-CatalogScrape.ps1). Feedback on the changes highlights several robustness improvements in the shared module, such as adding guard clauses to prevent ArgumentNullException crashes when calling .NET regex methods on null or empty content, resolving paths using PowerShell's provider path resolution to respect the current working directory, and avoiding unnecessary array wrapping that could introduce null elements.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +51 to +52
param([string]$Content)
$ids = [regex]::Matches($Content, "goToDetails\(['""]([a-f0-9\-]+)['""]\)") |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If $Content is null or empty, calling the .NET method [regex]::Matches directly will throw an ArgumentNullException and crash the script. Adding a guard clause to handle null or empty content makes the function robust.

    param([string]$Content)
    if ([string]::IsNullOrEmpty($Content)) { return @() }
    $ids = [regex]::Matches($Content, "goToDetails\(['""]([a-f0-9\-]+)['""]\)") |

Comment on lines +260 to +261
if (-not (Test-Path $DownloadPath)) { New-Item -ItemType Directory -Path $DownloadPath -Force | Out-Null }
$DownloadPath = [System.IO.Path]::GetFullPath($DownloadPath)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using [System.IO.Path]::GetFullPath($DownloadPath) resolves relative paths against the process's current working directory ([System.Environment]::CurrentDirectory), which is not updated when changing directories in PowerShell (e.g., via Set-Location or cd). This can cause the script to create the directory in the PowerShell current directory but then download/extract files to a completely different directory (like C:\Windows\System32\drivers), leading to unexpected behavior or failures.

To fix this, resolve the path using PowerShell's provider path resolution first, which correctly respects PowerShell's current working directory.

    $DownloadPath = $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($DownloadPath)
    if (-not (Test-Path $DownloadPath)) { New-Item -ItemType Directory -Path $DownloadPath -Force | Out-Null }

continue
}

$cabUrl = [regex]::Match($downloadPage.Content, 'https://[^''"<]+\.cab').Value

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If $downloadPage.Content is null, calling the .NET method [regex]::Match directly will throw an ArgumentNullException and crash the script. Using PowerShell's native -match operator is completely null-safe, more idiomatic, and avoids this potential crash.

            $cabUrl = if ($downloadPage.Content -match 'https://[^''"<]+\.cab') { $Matches[0] } else { $null }


# Expand each base query into its dual-query-union variants, then dedup.
$queries = @()
foreach ($baseQuery in @($Device.Queries)) { $queries += Expand-CsQuery $baseQuery }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Wrapping $Device.Queries in @(...) is an unnecessary anti-pattern in PowerShell. The foreach loop in PowerShell is already null-safe and handles both scalars and arrays natively. Wrapping $Device.Queries in @() actually creates an array with a single $null element if $Device.Queries is $null, which would cause the loop to execute once with $baseQuery = $null instead of safely skipping it.

            foreach ($baseQuery in $Device.Queries) { $queries += Expand-CsQuery $baseQuery }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant