Harden fuam lakehouse backup - #630
Open
DataCentricDave wants to merge 66 commits into
Open
Conversation
Bumps [launch-editor](https://github.com/vitejs/launch-editor) from 2.13.0 to 2.14.1. - [Commits](vitejs/launch-editor@v2.13.0...v2.14.1) --- updated-dependencies: - dependency-name: launch-editor dependency-version: 2.14.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [qs](https://github.com/ljharb/qs) to 6.15.2 and updates ancestor dependency [express](https://github.com/expressjs/express). These dependencies need to be updated together. Updates `qs` from 6.14.2 to 6.15.2 - [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md) - [Commits](ljharb/qs@v6.14.2...v6.15.2) Updates `express` from 4.22.1 to 4.22.2 - [Release notes](https://github.com/expressjs/express/releases) - [Changelog](https://github.com/expressjs/express/blob/v4.22.2/History.md) - [Commits](expressjs/express@v4.22.1...v4.22.2) --- updated-dependencies: - dependency-name: qs dependency-version: 6.15.2 dependency-type: indirect - dependency-name: express dependency-version: 4.22.2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
Update Python version in Deployment Notebook
Enrich Get-* output, add -Raw to 100% of getters, fix URL/retry bugs, add tooling
Enrichment / -Raw (the module-wide policy shift):
- Select-FabricResource flipped: default output is now enriched (full API object +
resolved-name NoteProperties + PSTypeName); -Raw returns the untouched response.
Previously this was inverted. Backfilled -Raw to every Get-* (167/167, 100%).
- Extended WorkspaceName/CapacityName/DatasetName enrichment + type to sub-resource
and admin getters that carry a resolvable id; scalar/definition/settings endpoints
get a clean -Raw passthrough. Enrichment only ADDS properties; nothing is dropped.
- Get-FabricWorkspaceRoleAssignment no longer trims the nested principal object.
Bug fixes:
- Add/Remove/Update-FabricConnectionRoleAssignment built /connections/roleAssignments/{id}
instead of /connections/{id}/roleAssignments (New-FabricAPIUri ordered -Subresource
before -ItemId). Added a -ResourceId alias and fixed the call sites.
- Invoke-FabricAPIRequest retry backoff threw an OverflowException ([int](Get-Date).Ticks)
when the API returned no Retry-After header. Extracted a tested Get-FabricRetryDelay
helper (Retry-After honored, exponential backoff + jitter, clamped 1..120s).
- Repaired corrupted comment-based help + dead $Items refs in Remove-FabricSharingLinks(Bulk).
Tests:
- New: ApiUriRegression, EnrichmentFlip, PropertyCompleteness (schema round-trip with
allOf/cross-file $ref resolution + negative control), Get-FabricRetryDelay,
Invoke-FabricAPIRequest retry behavior. Re-synced expected-param lists.
Tooling / docs:
- Update-FabricAPISpecsCache.ps1 now also caches the Power BI REST API spec and the
shared common definitions (Item base) for property-completeness validation.
- CLAUDE.md: enrichment-first policy, two-API model, PowerShell 7+ only (matches the
manifest); CHANGELOG updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@
- PSUseProcessBlockForPipelineCommand: wrapped 78 functions in process{} blocks. They
declare pipeline-bound parameters (ValueFromPipeline) but ran logic in a bare try/catch,
so multi-item pipeline input only processed the LAST item. Now every piped item is
processed - a real pipeline-correctness fix, not just a lint pass.
- Cleared all remaining PSScriptAnalyzer findings on source/Public (now 0): added
OutputType to the Resolve-* helpers + Export-FabricAdminDataflow; gave the empty
JSON-parse catch in Get-FabricOneLakeDataAccessRole a real statement; referenced the
param-set-only UseManagedIdentity in Set-FabricApiHeaders; added BOM to
New-FabricKQLDatabase/New-FabricWorkspace; trimmed trailing whitespace in Get-FabricDomain.
Suppressed a false-positive ShouldProcess warning on the private URI builder New-FabricAPIUri.
- Added an Author line to the 66 function files that lacked one.
- Removed the stale duplicate test Get-FabricExternalDataShares.Tests.ps1 (no plural function).
Reduces QA test-suite failures from 323 to the 165 functions still missing a unit test
(meaningful tests to follow). ScriptAnalyzer and Author gates are now fully green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…gments bug Tests: - Added behavior unit tests for all 80 Get-* functions that previously had none. Each asserts the exact constructed API endpoint + HTTP method, the default resolved-name enrichment + type decoration (or passthrough for definition/scalar endpoints), and that -Raw returns the untouched response. ~263 test cases, all green. Bug fix (surfaced while writing the tests): - New-FabricAPIUri never had a -Segments parameter, yet 34 functions called it with -Segments @(...) - every *Definition getter/updater built via segments plus the Domain functions. The call failed parameter binding, the error was swallowed by the surrounding try/catch, and the function returned nothing without ever calling the API. Added a -Segments parameter set that builds the path from an ordered segment list. Verified a definition getter now POSTs the correct .../getDefinition URL. Synced the New-FabricAPIUri param test. Reduces QA suite failures to 87 (all non-Get-* functions still lacking a unit test, out of this pass's Get-* scope) plus one full-run-only test-isolation flake. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pers Completes unit-test coverage for every previously test-less public function: - 16 New-*, 20 Remove-*, 33 Update-*, 5 Start-*, Stop/Set/Restore/Export/Invoke, 3 Add-*. State-changing functions assert the exact endpoint + HTTP method (invoked with -Confirm:$false) and include a -WhatIf-makes-no-API-call test where SupportsShouldProcess. - 3 Resolve-* helpers (CapacityIdFromWorkspace, DatasetName, GatewayName): resolved value, PSFramework caching, and id-fallback on not-found/error. - Update-*Definition functions POST to .../updateDefinition (exercising the just-fixed New-FabricAPIUri -Segments path). - The two module-visible helpers Add-FabricTypeName and Add-FabricDatasetProperties. With these, the QA "Should have a unit test for <fn>" gate is satisfied for all public functions. Remaining build,test failures are limited to a pre-existing cross-file test-isolation flake (Clear-FabricNameCache passes in isolation). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 'clear all cache entries at once' test asserted an exact count of 4 registered MicrosoftFabricMgmt.Cache.* entries, but PSFramework config is global to the session, so other test files leave additional cache entries during a full build,test run (it passed only in isolation). Relaxed to >= 4; the post-clear all-values-null assertion still proves the clear works. This was the sole remaining full-suite failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Complete the connection CRUD family (Get/Remove/RoleAssignment already existed):
- New-FabricConnection -> POST /connections
- Update-FabricConnection -> PATCH /connections/{connectionId}
The polymorphic (per connectivityType) request body is expressed with typed
-ConnectivityType/-GatewayId/-PrivacyLevel parameters plus -ConnectionDetails/
-CredentialDetails hashtables for the nested/credential parts. Both return the full
created/updated connection object (all API properties) enriched with a resolved
GatewayName and the MicrosoftFabric.Connection type; -Raw returns the untouched response.
SupportsShouldProcess (-WhatIf/-Confirm). Meaningful tests (11) + analyzer-clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
7 new commands for the generic item Job Scheduler API:
- New/Get/Update/Remove-FabricItemSchedule (.../items/{itemId}/jobs/{jobType}/schedules[/{scheduleId}])
- Start-FabricItemJob (POST .../jobs/{jobType}/instances)
- Get-FabricItemJobInstance (.../jobs/instances[/{jobInstanceId}])
- Stop-FabricItemJobInstance (POST .../jobs/instances/{jobInstanceId}/cancel)
Schedule -Configuration and job -ExecutionData are hashtable pass-throughs; getters
return the full object enriched with WorkspaceName + type and honor -Raw; state-changing
commands support -WhatIf/-Confirm. URIs built via New-FabricAPIUri -Segments.
Also fixed a New-FabricAPIUri binding bug: the mandatory [string[]] rejected a
null trailing element at binding time (an omitted -ScheduleId/-JobInstanceId), so the two
getters now build the segment list conditionally. Meaningful tests (27) + analyzer-clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New commands under /workspaces/{workspaceId}/git/... :
- Connect-FabricWorkspaceGit / Disconnect-FabricWorkspaceGit
- Initialize-FabricWorkspaceGitConnection
- Save-FabricWorkspaceGitCommit (commit workspace -> git)
- Update-FabricWorkspaceFromGit (update workspace <- git)
- Get-FabricWorkspaceGitStatus
- Get-FabricWorkspaceGitCredential / Update-FabricWorkspaceGitCredential
(Get-FabricWorkspaceGitConnection already existed.) Complex request bodies
(gitProviderDetails, commit mode/items, conflictResolution, credentials source) are
hashtable pass-throughs; state-changing commands support -WhatIf/-Confirm; getters honor
-Raw and add a MicrosoftFabric.* type. Meaningful tests (32) + analyzer-clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New commands under /deploymentPipelines/... : - Get/New/Update/Remove-FabricDeploymentPipeline - Get/Update-FabricDeploymentPipelineStage, Get-FabricDeploymentPipelineStageItem - Add/Remove-FabricDeploymentPipelineStageWorkspace (assign/unassign) - Invoke-FabricDeploymentPipelineDeploy - Get-FabricDeploymentPipelineOperation - Get/Add/Remove-FabricDeploymentPipelineRoleAssignment Deploy -Items/-Options are array/hashtable pass-throughs; state-changing commands support -WhatIf/-Confirm; getters honor -Raw and add a MicrosoftFabric.* type. Meaningful tests (55) + analyzer-clean. NOTE: spec marks create -Stages and deploy -TargetStageId as required but they are implemented optional (the API surfaces the error if omitted). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- DataPipeline definition (2): Get/Update-FabricDataPipelineDefinition - Warehouse restore points (5): Get/New/Update/Remove-FabricWarehouseRestorePoint, Restore-FabricWarehouseToRestorePoint - ML Model endpoint (8): Get/Update-FabricMLModelEndpoint, Get/Update-FabricMLModelEndpointVersion, Enable/Disable-FabricMLModelEndpointVersion (Disable -All => deactivateAll), Invoke-FabricMLModelEndpointScore, Invoke-FabricMLModelEndpointVersionScore - External Data Share (4): New-/Remove-FabricExternalDataShare (provider delete, distinct from revoke), Get-FabricExternalDataShareInvitation, Approve-FabricExternalDataShareInvitation Same pattern: full-object return + enrichment + -Raw + type, SupportsShouldProcess/-WhatIf on state-changing verbs, meaningful tests (71) + analyzer-clean. Complex/polymorphic bodies (external data share create/accept, ML endpoint scoring) use hashtable pass-throughs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full build,test green: 8525 passed, 0 failed, 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fills the remaining verified coverage gaps (9 new commands, 1 modified): - Workspace networking (8, preview): Get/Set network communication policy (PUT -IfMatch), outbound connection rules, outbound gateway rules, and Git outbound policy (PUT -IfMatch). Getters enrich WorkspaceName + type and honor -Raw; setters take hashtable bodies with -WhatIf/-Confirm. - Set-FabricOneLakeImmutabilityPolicy: POST modifyImmutabilityPolicy (-Scope DiagnosticLogs, -RetentionDays), -WhatIf/-Confirm, -Raw. - Get-FabricAdminRefreshable: -CapacityId now optional -> org-wide /admin/capacities/refreshables ($top defaulted); added -Expand. Full build,test green: 8626 passed / 0 failed / 1 skipped. Analyzer 0 on new files. Managed Private Endpoints already covered (no change needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Formatting backfill (display-only; enriched objects already carry every property): - Wire 10 standard item resource types (AnomalyDetector, DigitalTwinBuilder (+Flow), EventSchemaSet, GraphQuerySet, Map, MirroredAzureDatabricksCatalog, Ontology, OperationsAgent, UserDataFunction) into the shared Fabric item table + list views. - Add dedicated ConnectionView (name / connectivity type / gateway / id) and DeploymentPipelineView (name / description / id). - New structural test FormatViews.Tests.ps1 guards the wiring. Render-verified; full build,test green: 8631 passed / 0 failed / 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The script duplicated the $expectedParams block into every already-synced test file on a re-run: its param()-insert fallback fired whenever the regex replacement produced no change, which includes the already-correct case (253 files had both a param() block and the array). - Choose the branch by whether the block EXISTS, not by whether content changed; never fall through to insert when a block is already present. - Anchor the pattern on [ \t]* (not \s*) so replacement is byte-stable and re-runs are true no-ops. - Rename Replace- -> Set-ExpectedParamsInTest (approved verb) + suppress the Set/ShouldProcess smell; guard the main loop so the script is dot-sourceable. - New regression suite SyncExpectedParams.Tests.ps1 (3/3); idempotency also proven on the real tests/Unit tree (second run: 0 files changed). Full build,test green: 8634 passed / 0 failed / 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cache updater:
- Update-FabricAPISpecsCache.ps1 now auto-discovers every ./definitions/{name}.json
a swagger $refs and downloads it as {spec}.{name}.definitions.json (15 nested files
across platform/admin/eventstream, 0 failures). This makes the body/response schemas
for networking, connections, deployment pipelines, git integration and external data
sharing durably resolvable during validation instead of relying on ad-hoc fetches.
Version:
- Bumped source manifest ModuleVersion 1.0.8 -> 1.1.0 (minor bump for the ~59 new
commands this cycle). Build emits output/module/.../1.1.0/ and the clean task drops
the old 1.0.8 folder, so freshly built dev modules sort above the stale locally
installed 1.0.9 and win module auto-resolution.
Full build,test green at 1.1.0: 8634 passed / 0 failed / 1 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Job-type-in-path modernization (legacy ?jobType= query form still works;
path form is the documented shape):
- Start-FabricLakehouseRefreshMaterializedLakeView -> .../jobs/RefreshMaterializedLakeViews/instances
- Start-FabricLakehouseTableMaintenance -> .../jobs/TableMaintenance/instances
- Start-FabricSparkJobDefinitionOnDemand -> .../sparkJobDefinitions/{id}/jobs/sparkjob/instances
(also removed a dead undefined $bodyJson reference). Each gains behavior
tests asserting the new URL and that the query form is gone.
Format views (backfill pass 2): WarehouseSnapshot folded into the shared
item views; new ItemJobInstanceView / ItemScheduleView / LakehouseTableView /
WarehouseRestorePointView (render-verified). Single-object/status/scalar types
stay on default list per the "distinct table only" rule. FormatViews test extended.
Also revert the CHANGELOG heading to the literal "## [Unreleased]" that the
prior commit changed to "## [1.1.0] - Unreleased" -- the latter broke the
keepachangelog QA gate and the release-notes stamping (parses the version
line's date). Sampler stamps the version from [Unreleased] at release time.
Full build,test green: 8646 passed / 0 failed / 1 skipped, 15 tasks 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New scripts/New-FabricPropertyMap.ps1 (spec-driven, repeatable) generates docs/api-property-map.md: per-resource tables of every API response-schema property (name + type + nested $ref) against what the module returns. Confirms the enrichment-first contract - every top-level API property is returned by default (verified by PropertyCompleteness.Tests.ps1), -Raw is untouched, and default output adds the resolved-name NoteProperties. The doc's "High-value targets" table is the flattening backlog: nested scalars users most want (connection strings, service URIs, OneLake paths, SQL FQDN / restore points, provisioning state) that we should surface as flat top-level fields next - e.g. properties.sqlEndpointProperties.connectionString -> SqlEndpointConnectionString. Generic id/displayName metadata objects (defaultIdentity/sensitivityLabel/tags) are summarised once as lower priority. 26 resources mapped, 10 high-value flatten targets. todo.md updated with the next-work checklist. No module/test sources changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements the docs/api-property-map.md flattening backlog. New private Add-FabricFlattenedProperty surfaces scalars buried under an item's nested `properties` object as flat top-level NoteProperties, called from the enrich branch of Select-FabricResource so every item getter that uses it gains the fields at once. It never runs on -Raw (that block is skipped), leaves the nested object intact, is guarded per-field and idempotent. Surfaced fields include: ConnectionString, OneLakeTablesPath/FilesPath/RootPath, DefaultSchema, QueryServiceUri, IngestionServiceUri, ServerFqdn, DatabaseName, Earliest/LatestRestorePoint, BackupRetentionDays, SqlEndpointConnectionString (+ Id, ProvisioningStatus) from properties.sqlEndpointProperties, and PublishState from properties.publishDetails.state. Tests: Private/Add-FabricFlattenedProperty.Tests.ps1 (7 cases) + EnrichmentFlip default-flattens / -Raw-does-not cases. Also remove a pre-existing orphan test (New-FabricNotebookNEW.Tests.ps1) that referenced a non-existent function and caused a silent Pester container- discovery failure. Full build,test green: 8637 passed / 0 failed / 1 skipped, 15 tasks 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audited every enriching Get-* that does not route through Select-FabricResource against the specs. The only one carrying a flattenable item `properties` object is Get-FabricWarehouseSnapshot (properties.connectionString / parentWarehouseId / snapshotDateTime) - wired Add-FabricFlattenedProperty into its enrich path so it gains ConnectionString / ParentWarehouseId / SnapshotDateTime flat fields (skipped on -Raw). Added flatten + -Raw behavior tests. Every other manual-enrichment getter is a sub-resource/status (users, datasources, livy sessions, tables, mirroring/git statuses, networking policies, ML endpoints) with no item `properties` object, so there is nothing to flatten - the sweep is complete. Full build,test green: 8639 passed / 0 failed / 1 skipped, 15 tasks 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The -Filter parameter never worked: the Fabric admin GET /admin/workspaces endpoint does not support OData query options ($filter/$top/$skip/$orderby), so those values were silently ignored by the server (confirmed against the official REST API reference). - Remove the four phantom parameters (-Filter/-Top/-Skip/-OrderBy). - Add the genuinely-supported -EncryptionStatus (auto-sets -Include encryption, since the API only applies the filter when encryption is included) and -Include. - Fix -WorkspaceName: it returned nothing because results were re-filtered client-side on `displayName` while the admin API returns `name`. Filtering is now server-side via the `name` query param; the client-side re-filter is removed. - Refresh comment-based help / examples / notes to document the real supported query params and point to client-side Where-Object/Select-Object for the rest. New behavior tests assert the named filters hit the query string, -EncryptionStatus auto-includes encryption, -WorkspaceName no longer drops rows, and the OData parameters are gone. Full build,test green: 8643 passed / 0 failed / 1 skipped, 15 tasks 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings datasource up to date with main (was 135 commits behind; main advanced to 1.0.9). Only tools/MicrosoftFabricMgmt conflicted; all source .ps1 merged cleanly. Resolutions: - source manifest: keep ModuleVersion 1.1.0 (FunctionsToExport is regenerated from source/Public at build). - CHANGELOG.md: union - kept our [Unreleased] entries and folded in main's New-FabricSemanticModel/New-FabricReport -FolderId (Added) and helper-pattern modernization (Changed). - output/**: regenerated by a fresh build (dropped main's stale 1.0.9/1.0.6 version folders; output now only carries 1.1.0). Full build,test green post-merge: 8651 passed / 0 failed / 1 skipped, 15 tasks 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The module's authentication command is now Connect-FabricAccount, matching
Azure/Fabric tooling conventions and the name already referenced throughout the
help. Set-FabricApiHeaders is retained as an exported alias (via [Alias()] on the
function, auto-exported by ModuleBuilder) so existing scripts keep working.
- git mv source/Public/Utils/Set-FabricApiHeaders.ps1 -> Connect-FabricAccount.ps1
and the test to Connect-FabricAccount.Tests.ps1; renamed docs md.
- Bulk-updated every Set-FabricApiHeaders reference across source .ps1, docs and
README to Connect-FabricAccount (help text, user-facing "please run ..." messages,
module banner). The only literals left are the [Alias('Set-FabricApiHeaders')]
attribute and its NOTES mention.
- New tests assert the alias exists, resolves to Connect-FabricAccount, and is in
the module's ExportedAliases.
Full build,test green: 8654 passed / 0 failed / 1 skipped, 15 tasks 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mpo/fabric-toolbox into fat-aws-databricks-support
Bumps [morgan](https://github.com/expressjs/morgan) from 1.10.1 to 1.11.0. - [Release notes](https://github.com/expressjs/morgan/releases) - [Changelog](https://github.com/expressjs/morgan/blob/master/HISTORY.md) - [Commits](expressjs/morgan@1.10.1...1.11.0) --- updated-dependencies: - dependency-name: morgan dependency-version: 1.11.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [websocket-driver](https://github.com/faye/websocket-driver-node) from 0.7.4 to 0.7.5. - [Changelog](https://github.com/faye/websocket-driver-node/blob/main/CHANGELOG.md) - [Commits](faye/websocket-driver-node@0.7.4...0.7.5) --- updated-dependencies: - dependency-name: websocket-driver dependency-version: 0.7.5 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [axios](https://github.com/axios/axios) from 1.16.0 to 1.18.0. - [Release notes](https://github.com/axios/axios/releases) - [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md) - [Commits](axios/axios@v1.16.0...v1.18.0) --- updated-dependencies: - dependency-name: axios dependency-version: 1.18.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
…rn/accelerators/mirror-lakehouse/client/launch-editor-2.14.1 build(deps-dev): bump launch-editor from 2.13.0 to 2.14.1 in /accelerators/mirror-lakehouse/client
Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 4.1.1 to 4.3.0. - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@4.1.1...4.3.0) --- updated-dependencies: - dependency-name: js-yaml dependency-version: 4.3.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
MicrosoftFabricMgmt 1.1.0 — ~60 new commands, enrichment/-Raw, property flattening, admin-workspace filter fix
…rn/accelerators/mirror-lakehouse/server/morgan-1.11.0 build(deps): bump morgan from 1.10.1 to 1.11.0 in /accelerators/mirror-lakehouse/server
…rn/accelerators/mirror-lakehouse/server/multi-f792d6d6d9 build(deps): bump qs and express in /accelerators/mirror-lakehouse/server
…rn/accelerators/mirror-lakehouse/server/axios-1.18.0 build(deps): bump axios from 1.16.0 to 1.18.0 in /accelerators/mirror-lakehouse/server
…port Fabric Assessment Tool - AWS Databricks support
Bumps [shell-quote](https://github.com/ljharb/shell-quote) from 1.8.4 to 1.10.0. - [Changelog](https://github.com/ljharb/shell-quote/blob/main/CHANGELOG.md) - [Commits](ljharb/shell-quote@v1.8.4...v1.10.0) --- updated-dependencies: - dependency-name: shell-quote dependency-version: 1.10.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
…rn/accelerators/mirror-lakehouse/client/websocket-driver-0.7.5 build(deps-dev): bump websocket-driver from 0.7.4 to 0.7.5 in /accelerators/mirror-lakehouse/client
Bumps [body-parser](https://github.com/expressjs/body-parser) from 1.20.5 to 1.20.6. - [Release notes](https://github.com/expressjs/body-parser/releases) - [Changelog](https://github.com/expressjs/body-parser/blob/master/HISTORY.md) - [Commits](expressjs/body-parser@1.20.5...1.20.6) --- updated-dependencies: - dependency-name: body-parser dependency-version: 1.20.6 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
…rn/tools/FabricDataFactoryMigrationAssistant/js-yaml-4.3.0 build(deps-dev): bump js-yaml from 4.1.1 to 4.3.0 in /tools/FabricDataFactoryMigrationAssistant
Bumps [axios](https://github.com/axios/axios) from 1.16.0 to 1.18.0. - [Release notes](https://github.com/axios/axios/releases) - [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md) - [Commits](axios/axios@v1.16.0...v1.18.0) --- updated-dependencies: - dependency-name: axios dependency-version: 1.18.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [fast-uri](https://github.com/fastify/fast-uri) from 3.1.2 to 3.1.5. - [Release notes](https://github.com/fastify/fast-uri/releases) - [Commits](fastify/fast-uri@v3.1.2...v3.1.5) --- updated-dependencies: - dependency-name: fast-uri dependency-version: 3.1.5 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
Fabric Assessment Tool - Parallelize databricks client
feat: add Lakehouse copy/move utility
…rn/accelerators/mirror-lakehouse/server/body-parser-1.20.6 build(deps): bump body-parser from 1.20.5 to 1.20.6 in /accelerators/mirror-lakehouse/server
…rn/accelerators/mirror-lakehouse/shell-quote-1.10.0 build(deps-dev): bump shell-quote from 1.8.4 to 1.10.0 in /accelerators/mirror-lakehouse
…rn/accelerators/mirror-lakehouse/client/axios-1.18.0 build(deps): bump axios from 1.16.0 to 1.18.0 in /accelerators/mirror-lakehouse/client
…rn/accelerators/mirror-lakehouse/client/fast-uri-3.1.5 build(deps-dev): bump fast-uri from 3.1.2 to 3.1.5 in /accelerators/mirror-lakehouse/client
Adds retry/snapshot-safe backups and safer cleanup of old and empty backup folders.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Pull Request (PR) description
The FUAM Lakehouse Backup notebook previously copied tables and files straight from the live lakehouse with a single
fastcpcall and cleaned up old backups with an ad hocmssparkutilsfolder walk. A transient file error or a concurrent write/OPTIMIZE on a live table could fail the whole run, and a crashed run could leave stray snapshot or empty folders behind.Added
safe_backuphelper that snapshots tables via a Delta shallow clone before copying (so a live write/OPTIMIZE can't cause a mid-copy failure), retries on transientfastcperrors, and always cleans up its snapshot metadata.remove_empty_dirshelper and a notebook cell to remove empty folders left behind after old backups are pruned.Changed
cleanup_old_backupshelper (skips non-date folders instead of assuming everything under the backup root is a dated folder).BACKUP_ROOT,SNAPSHOT_ROOT,DATE_FOLDER_DEPTH,EMPTY_DIR_SCAN_DEPTH) into the Parameters cell.Task list
build.ps1 -ResolveDependency -Tasks build, test).