Unify catalog driver scrapers into a shared module + data tables#2
Conversation
Replace six drifted copy-paste Get-*Drivers.ps1 scrapers with a shared catalogscrape/CatalogScrape.psm1 engine, per-vendor .psd1 device tables, and thin shims (still auto-discovered by build_win11pxe.ps1). Correctness fixes (verified against the live Microsoft Update Catalog): - E810 and Marvell AQC111U-USB returned 0 results and were silently skipped every run. E810: the "Windows 11" suffix zeroes a datacenter NIC query. AQC111U: VID_1D6A is the PCIe vendor id, not USB -> use the first-party VID_2ECA&PID_C101. - Selection is now highest-[version]-first (catalog date as tiebreak), fixing Realtek-USB (stale catalog date), Intel re-release, and MediaTek stale preferred-branch, which all picked an older driver than offered. - Add per-request timeouts, culture-invariant date parsing, tolerant version parsing, pnputil /subdirs, and broken-extract cleanup. New coverage: - Intel X520 (82599), Broadcom NetXtreme-E/C (10-100G), and QLogic FastLinQ (41xxx/45xxx) folded into the Marvell module (FastLinQ is Marvell-owned). Durability (25-row cap): - Dual-query union (Expand-CsQuery): HWID queries are sampled from both the bare and "+Windows 11" relevance windows, deduped, highest version wins. The catalog caps results at 25 relevance rows and rejects scripted sort/pagination postbacks, so this is the dependency-free way to beat the cap (replaces the per-device AppendOsToken suffix guess). Adds catalogscrape/Test-CatalogScrape.ps1 (68 network-free unit tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request refactors several vendor-specific driver scraping scripts into thin shims that utilize a new, shared PowerShell module (CatalogScrape.psm1) driven by vendor-specific .psd1 configuration files, and introduces a unit test suite (Test-CatalogScrape.ps1). Feedback on the changes highlights several robustness improvements in the shared module, such as adding guard clauses to prevent ArgumentNullException crashes when calling .NET regex methods on null or empty content, resolving paths using PowerShell's provider path resolution to respect the current working directory, and avoiding unnecessary array wrapping that could introduce null elements.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| param([string]$Content) | ||
| $ids = [regex]::Matches($Content, "goToDetails\(['""]([a-f0-9\-]+)['""]\)") | |
There was a problem hiding this comment.
If $Content is null or empty, calling the .NET method [regex]::Matches directly will throw an ArgumentNullException and crash the script. Adding a guard clause to handle null or empty content makes the function robust.
param([string]$Content)
if ([string]::IsNullOrEmpty($Content)) { return @() }
$ids = [regex]::Matches($Content, "goToDetails\(['""]([a-f0-9\-]+)['""]\)") |
| if (-not (Test-Path $DownloadPath)) { New-Item -ItemType Directory -Path $DownloadPath -Force | Out-Null } | ||
| $DownloadPath = [System.IO.Path]::GetFullPath($DownloadPath) |
There was a problem hiding this comment.
Using [System.IO.Path]::GetFullPath($DownloadPath) resolves relative paths against the process's current working directory ([System.Environment]::CurrentDirectory), which is not updated when changing directories in PowerShell (e.g., via Set-Location or cd). This can cause the script to create the directory in the PowerShell current directory but then download/extract files to a completely different directory (like C:\Windows\System32\drivers), leading to unexpected behavior or failures.
To fix this, resolve the path using PowerShell's provider path resolution first, which correctly respects PowerShell's current working directory.
$DownloadPath = $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($DownloadPath)
if (-not (Test-Path $DownloadPath)) { New-Item -ItemType Directory -Path $DownloadPath -Force | Out-Null }
| continue | ||
| } | ||
|
|
||
| $cabUrl = [regex]::Match($downloadPage.Content, 'https://[^''"<]+\.cab').Value |
There was a problem hiding this comment.
If $downloadPage.Content is null, calling the .NET method [regex]::Match directly will throw an ArgumentNullException and crash the script. Using PowerShell's native -match operator is completely null-safe, more idiomatic, and avoids this potential crash.
$cabUrl = if ($downloadPage.Content -match 'https://[^''"<]+\.cab') { $Matches[0] } else { $null }
|
|
||
| # Expand each base query into its dual-query-union variants, then dedup. | ||
| $queries = @() | ||
| foreach ($baseQuery in @($Device.Queries)) { $queries += Expand-CsQuery $baseQuery } |
There was a problem hiding this comment.
Wrapping $Device.Queries in @(...) is an unnecessary anti-pattern in PowerShell. The foreach loop in PowerShell is already null-safe and handles both scalars and arrays natively. Wrapping $Device.Queries in @() actually creates an array with a single $null element if $Device.Queries is $null, which would cause the loop to execute once with $baseQuery = $null instead of safely skipping it.
foreach ($baseQuery in $Device.Queries) { $queries += Expand-CsQuery $baseQuery }
Summary
Unifies the six Microsoft Update Catalog driver scrapers into one
catalogscrape/CatalogScrape.psm1engine + per-vendor.psd1device tables + thinGet-*Drivers.ps1shims (still auto-discovered bybuild_win11pxe.ps1). ~120 lines of copy-pasted, drifted boilerplate collapse to one place.Correctness fixes (live-verified against the catalog)
"Windows 11"suffix zeroes a datacenter-NIC query. Now bare HWID (0 → 4 results).VID_1D6Ais the PCIe vendor id; the real USB id isVID_2ECA&PID_C101(0 → 20 results, Aquantia v1.8.0.0). Dropped the dead TRENDnet/ASIX entries.[version]-first (catalog date as tiebreak), fixing Realtek-USB (stale 2016 catalog date), Intel re-release, and MediaTek stale preferred-branch — all of which picked an older driver than the catalog offered.pnputil /subdirs, and broken-extract cleanup.New coverage
25-row cap
Expand-CsQuery): HWID queries are sampled from both the bare and+ Windows 11relevance windows, deduped, highest version wins (replaces the per-deviceAppendOsTokensuffix guess). The catalog caps results at 25 relevance rows and rejects scripted sort/pagination postbacks (confirmed for both the MSCatalog module and a hand-rolled VIEWSTATE replay), so two windows is the durable, dependency-free option.Validation
catalogscrape/Test-CatalogScrape.ps1), lint clean, plus live end-to-end engine checks (Linux PowerShell 7) for every fix.expand.exe/pnputil) are copied verbatim from the originals (can't run on Linux).🤖 Generated with Claude Code