Skip to content

Commit 89f2ad7

Browse files
authored
Refactor dav: Adopt two-layer architecture to match other storage providers (#98)
* refactor(dav): Adopt two-layer architecture to match other storage providers Restructure the DAV client to align with S3, Azure, GCS, and AliOSS provider patterns in storage-cli. This is the first step in modernizing the DAV implementation, with advanced features to follow in subsequent PRs. **Structural Changes:** - Split client into two layers: - `client.go`: High-level DavBlobstore implementing storage.Storager - `storage_client.go`: Low-level StorageClient for HTTP/WebDAV operations - Remove cmd/ and app/ layers (now integrated directly via storage/factory.go) - Remove test_assets/ directory (replaced by Docker-based testing) - Remove old fakes/fake_client.go (replaced by clientfakes/fake_storage_client.go) - Add helpers.go for shared utilities (getCertPool) **Stub Methods (to be implemented in follow-up PRs):** - List: List blobs with optional prefix filter - Copy: Server-side blob copying via WebDAV COPY - DeleteRecursive: Delete all blobs matching prefix - Properties: Retrieve blob metadata - EnsureStorageExists: Initialize WebDAV directory structure **Path Handling:** - Remove automatic 2-character path fan-out. Previously "abcdef..." was stored as "ab/cd/abcdef...". Now object IDs are passed through unchanged. - This aligns with other storage-cli providers (S3/Azure/GCS/AliOSS) and matches existing CCNG usage (BaseClient#partitioned_key already handles partitioning before calling storage-cli). - Callers are responsible for any path layout or partitioning strategy. **Testing:** - Update tests to use mocked StorageClient with counterfeiter - Add comprehensive test coverage for Exists (found, not found, error cases) - Add docker-compose.yml for local WebDAV testing - Update README with breaking change notice and usage examples **Code Quality:** - Align logging style with other providers (lowercase, Debug for success messages) - All linter checks pass (golangci-lint, go vet, gofmt) - 100% test pass rate **Breaking Changes:** - Removed automatic path partitioning. Object IDs are used exactly as provided. - Callers must include any desired directory structure in the object ID. - See dav/README.md "Path Handling Contract" section for migration guidance. * Update readme * change log level
1 parent 4ee94c5 commit 89f2ad7

33 files changed

Lines changed: 843 additions & 1853 deletions

dav/README.md

Lines changed: 69 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -6,28 +6,78 @@ WebDAV client implementation for the unified storage-cli tool. This module provi
66

77
For general usage and build instructions, see the [main README](../README.md).
88

9+
## Path Handling Contract
10+
11+
**IMPORTANT:** Callers are responsible for any path layout or partitioning strategy. The DAV driver does not transform object IDs.
12+
13+
Object IDs are used exactly as provided:
14+
- Simple paths: `my-blob-id` → stored as `my-blob-id`
15+
- Nested paths: `ab/cd/my-blob-id` → stored as `ab/cd/my-blob-id`
16+
17+
**Breaking change in v0.0.8+:** The driver no longer applies automatic 2-character path fan-out (e.g., `abcdef...` was previously stored as `ab/cd/abcdef...` in v0.0.7 and earlier). Callers must now include any desired directory structure in the object ID itself.
18+
19+
This behavior aligns with other storage-cli providers (S3, Azure, GCS, AliOSS), which also pass keys through unchanged.
20+
921
## DAV-Specific Configuration
1022

11-
The DAV client requires a JSON configuration file with WebDAV endpoint details and credentials.
23+
The DAV client requires a JSON configuration file with the following structure:
24+
25+
``` json
26+
{
27+
"endpoint": "<string> (required - WebDAV server URL)",
28+
"user": "<string> (optional - for Basic Auth)",
29+
"password": "<string> (optional - for Basic Auth)",
30+
"retry_attempts": <uint> (optional - default: 3),
31+
"tls": {
32+
"cert": {
33+
"ca": "<string> (optional - PEM-encoded CA certificate)"
34+
}
35+
},
36+
"secret": "<string> (optional - required for pre-signed URLs)"
37+
}
38+
```
1239

1340
**Usage examples:**
1441
```bash
15-
# Upload an object
16-
storage-cli -s dav -c dav-config.json put local-file.txt remote-object
42+
# Upload a file to WebDAV
43+
storage-cli -s dav -c dav-config.json put local-file.txt remote-blob
1744

18-
# Fetch an object
19-
storage-cli -s dav -c dav-config.json get remote-object local-file.txt
45+
# Download a file from WebDAV
46+
storage-cli -s dav -c dav-config.json get remote-blob local-file.txt
2047

21-
# Delete an object
22-
storage-cli -s dav -c dav-config.json delete remote-object
48+
# Check if blob exists
49+
storage-cli -s dav -c dav-config.json exists remote-blob
2350

24-
# Check if an object exists
25-
storage-cli -s dav -c dav-config.json exists remote-object
51+
# Delete a blob
52+
storage-cli -s dav -c dav-config.json delete remote-blob
2653

27-
# Generate a signed URL (e.g., GET for 1 hour)
28-
storage-cli -s dav -c dav-config.json sign remote-object get 60s
54+
# Generate a pre-signed URL (requires secret in config)
55+
storage-cli -s dav -c dav-config.json sign remote-blob get 1h
2956
```
3057

58+
## Features
59+
60+
### Basic Operations (Available)
61+
- **Put** - Upload files to WebDAV server
62+
- **Get** - Download files from WebDAV server
63+
- **Delete** - Delete individual blobs
64+
- **Exists** - Check if a blob exists
65+
- **Sign** - Generate pre-signed URLs with HMAC-SHA256
66+
67+
### Advanced Operations (Coming Soon)
68+
The following operations will be available in future releases:
69+
- **List** - List all blobs or filter by prefix
70+
- **Copy** - Server-side blob copying via WebDAV COPY method
71+
- **DeleteRecursive** - Delete all blobs matching a prefix
72+
- **Properties** - Retrieve blob metadata (ContentLength, ETag, LastModified)
73+
- **EnsureStorageExists** - Initialize WebDAV directory structure
74+
75+
### Automatic Retry Logic
76+
All operations automatically retry on transient errors. Default is 3 retry attempts, configurable via `retry_attempts` in config.
77+
78+
### TLS/HTTPS Support
79+
Supports HTTPS connections with custom CA certificates for internal or self-signed certificates.
80+
3181
## Pre-signed URLs
3282

3383
The `sign` command generates a pre-signed URL for a specific object, action, and duration.
@@ -44,6 +94,13 @@ The generated URL format:
4494

4595
### Unit Tests
4696
Run unit tests from the repository root:
97+
4798
```bash
48-
ginkgo --cover -v -r ./dav/...
99+
go test ./dav/client/...
49100
```
101+
102+
Or using ginkgo:
103+
```bash
104+
ginkgo --cover -v -r ./dav/client
105+
```
106+

dav/app/app.go

Lines changed: 0 additions & 80 deletions
This file was deleted.

dav/app/app_suite_test.go

Lines changed: 0 additions & 13 deletions
This file was deleted.

dav/app/app_test.go

Lines changed: 0 additions & 166 deletions
This file was deleted.

0 commit comments

Comments
 (0)