SPSUserSync: v1.3.0 — optional parallel AD resolution (RunspacePool, opt-in)#15
Merged
Conversation
Proof of concept for parallelizing the AD-resolution part of SPSyncUserInfoList.ps1 on large multi-forest farms. Windows PowerShell 5.1 has no ForEach-Object -Parallel, so this uses a RunspacePool directly, in the spirit of SPSWakeUp's Invoke-SPSWebRequest. Only AD resolution is parallelized. SPSite/SPWeb objects are not thread-safe and must not cross runspaces, so the intended integration is three phases: 1. sequential (SharePoint thread): walk webs, collect UNIQUE user logins 2. parallel: Resolve-SPSADUserBatch resolves those logins against AD 3. sequential (SharePoint thread): build JSON + Set-SPUser / Remove-SPUser This commit delivers phase 2 only; the entry-point scripts are NOT touched yet. New module functions (exported; now 15): - Get-SPSThrottleLimit: degree-of-parallelism heuristic from [Environment]::ProcessorCount (cap 10 at 8+ logical CPUs, else 2x, min 2). Cross-platform, so it is testable off a SharePoint server. - Resolve-SPSADUserBatch: resolves a batch of unique logins through a RunspacePool and returns one record per login (DisplayName, FirstName, LastName, Email, Country, Location, Resolved, Error). The default worker calls Get-SPSADUser (each runspace imports SPSUserSync.Common via the InitialSessionState) and projects exactly the same attributes as the current sequential loop, so the JSON stays identical. A failing login is isolated (Resolved=$false + Error) instead of aborting the batch. An injectable -ResolveScript lets tests exercise the parallel plumbing without a live directory. Tests (tests/Resolve-SPSADUserBatch.Tests.ps1, cross-platform via a mock worker): one result per login, no loss/duplication, attribute projection, empty input, per-login error isolation, and a concurrency wall-time assertion. Measured locally (PowerShell 7, mock 100ms LDAP latency, 40 logins, throttle 10): sequential ~4090 ms vs parallel ~490 ms (~8.4x). Validation: PSScriptAnalyzer clean, Pester 59/59. NOT for main as-is: this is a spike on feature/parallel-ad-resolution for review. Next, if we proceed: wire phases 1+3 into SPSyncUserInfoList.ps1 behind an opt-in setting (e.g. ParallelADResolution / MaxParallelADQueries), since the win only matters on big farms and adds per-runspace module-import overhead.
Wires the parallel RunspacePool resolver into SPSyncUserInfoList.ps1 behind a
new opt-in setting, and refactors Get-SPSUniqueUsers into three clean phases so
the SharePoint work and the AD work no longer interleave.
Get-SPSUniqueUsers is now:
1. Phase 1 (sequential, SharePoint): walk webs, apply exclusions, run the
Set-SPUser/Remove-SPUser cleanup (unchanged), and collect ONE snapshot per
unique login (first occurrence: its SP DisplayName + Email feed the JSON
fallbacks). Dedup uses an ordinal OrderedDictionary, matching the previous
case-sensitive Array.Contains behavior exactly.
2. Phase 2 (AD): resolve the unique logins. Parallel via Resolve-SPSADUserBatch
when ParallelADResolution = $true, else a sequential loop. Both go through
ConvertTo-SPSUserRecord, so the projected attributes are identical.
3. Phase 3 (sequential, no SharePoint): build the JSON records with the same
DisplayName/email fallbacks as before.
Each unique login is now resolved exactly once (previously once per web it
appeared in), which also speeds up the default sequential mode on farms where
users span many sites. The user-removal walk is untouched and still per web.
New settings (sync-settings.example.psd1, both default-safe):
- ParallelADResolution (default $false)
- MaxParallelADQueries (default 0 = auto via Get-SPSThrottleLimit)
New public helper:
- ConvertTo-SPSUserRecord — single Get-SPSADUser -> record projection shared by
the sequential path and the parallel worker. Resolve-SPSADUserBatch's default
worker now calls it, removing the duplicated extraction. Module exports 16.
Tests (tests/ConvertTo-SPSUserRecord.Tests.ps1, cross-platform via a fake AD
object): full projection, co/l upper-casing, the "First Last" displayName
fallback, all-empty attrs, and the null (unresolved) entry; plus a test that the
parallel worker yields the same shape.
Proven locally: with a shared fake directory, the sequential and parallel paths
produce byte-for-byte identical records (DisplayName/First/Last/Email/Country/
Location/Resolved), including the empty-attrs and not-in-AD cases. PSScriptAnalyzer
clean, Pester 65/65.
NOT for main as-is: continues the feature/parallel-ad-resolution spike. Needs a
real-farm run (toggle ParallelADResolution) before a 1.3.0 release. Docs updated
in CHANGELOG [Unreleased] and wiki/Configuration.md.
Bump the module manifest to 1.3.0, date the CHANGELOG [1.3.0] section, and rewrite RELEASE-NOTES.md (the GitHub Release body) for this version. Highlights: optional parallel AD resolution (RunspacePool, opt-in via ParallelADResolution, off by default) for large multi-forest farms, with a byte-for-byte identical JSON verified on a SharePoint Subscription Edition farm. Refs #14
Pester Test Results65 tests 65 ✅ 2s ⏱️ Results for commit c69e69e. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request (PR) description
Ships SPSUserSync v1.3.0. Adds optional parallel Active Directory
resolution to
SPSyncUserInfoList.ps1, for large multi-forest farms where theper-user LDAP round-trip dominates the runtime. Opt-in and off by default;
the generated JSON is byte-for-byte identical whether parallel resolution is
on or off — verified on a SharePoint Subscription Edition farm
(
Get-FileHashmatch between a sequential and a parallel run).Windows PowerShell 5.1 has no
ForEach-Object -Parallel, so the resolver uses aRunspacePool directly, in the spirit of SPSWakeUp's
Invoke-SPSWebRequest. Onlythe AD resolution runs in parallel; SPSite/SPWeb objects are not thread-safe and
stay on the SharePoint thread.
Added
ParallelADResolution = $true(newsetting, default
$false), the unique user logins are resolved against ADconcurrently via a RunspacePool.
MaxParallelADQueriessets the degree ofparallelism (0 = auto from the CPU count). Two new public helpers:
Resolve-SPSADUserBatch(resolver) andGet-SPSThrottleLimit(CPU default).~8x speedup on a 40-user mock with 100 ms LDAP latency.
ConvertTo-SPSUserRecord([Feature]: Parallel AD resolution for large farms (RunspacePool, opt-in) #14) — the singleGet-SPSADUser-> recordprojection shared by both the sequential path and the parallel worker, which
guarantees the identical JSON.
Changed
SPSyncUserInfoList.ps1resolves each unique login against AD exactly once(previously once per web it appeared in), by splitting
Get-SPSUniqueUsersinto three phases: (1) SharePoint walk + exclusions +
Set-SPUser/Remove-SPUsercleanup + unique-snapshot collection, (2) AD resolution (parallel or
sequential), (3) JSON building. This also speeds up the default sequential
mode. The user-removal walk is unchanged and still runs per web. ([Feature]: Parallel AD resolution for large farms (RunspacePool, opt-in) #14)
Upgrade notes
behavior. Add them to
sync-settings.psd1only to enable parallel resolution.Validation
Invoke-Pester ./tests-> 65/65 passed (new cross-platform tests forResolve-SPSADUserBatch,Get-SPSThrottleLimit,ConvertTo-SPSUserRecord)Invoke-ScriptAnalyzer ./src -Recurse -Settings ./PSScriptAnalyzerSettings.psd1-> cleanempty-attribute and not-in-AD cases).
unique resolved in parallel;
Get-FileHashof the JSON identical to thesequential run.
This Pull Request (PR) fixes the following issues
Task list
file CHANGELOG.md. Entry should say what was changed and how that
affects users (if applicable), and reference the issue being resolved
(if applicable).