Object Storage and S3 API Guide

Object Storage and S3 API Guide

Language: English | 한국어

1. CB-Spider Object Storage Overview

• CB-Spider can manage multi-cloud Object Storage in a unified manner using AWS S3 standard methods and compatible APIs.
• Users can create Buckets and store/manage Objects (files) in created Buckets by assigning unique Keys.
• Users can create and manage multiple Buckets, and each Bucket can store and manage multiple Key:Object pairs.

2. CB-Spider Object Storage Features

• Object Storage Features

  • Bucket Management

    • Create, retrieve, delete, and manage the status of Buckets, which are the basic unit of object storage
    • Provides bucket existence check and bucket location information retrieval

  • Object Management

    • Basic data management functions to upload, download, retrieve, and delete objects within a bucket
    • Supports file-based and form-based object upload
    • Object metadata retrieval and batch deletion of multiple objects

  • Multipart Upload Management

    • Multipart Upload management for uploading large objects by splitting them into multiple parts
    • Supports multipart upload initialization, part upload, completion, and abort
    • Ability to list in-progress multipart operations and individual parts

  • Versioning Management

    • Object versioning management at the bucket level
    • Retrieve and enable/disable versioning settings
    • List all versions of an object and delete specific versions

  • CORS Management

    • Cross-Origin Resource Sharing (CORS) policy management for buckets
    • Provides CORS configuration retrieval, registration, and deletion

  • Special (PreSigned URL, Force Delete, and Metadata Management)

    • PreSigned URL generation allowing object access for a limited time without authentication
      • PreSigned URL for object download
      • PreSigned URL for object upload
    • Empty Bucket function to forcibly delete all objects in a bucket
    • Force Delete function to forcibly delete a bucket and its contained objects
    • CB-Spider metadata management functions
      • Bucket registration (Register)
      • Bucket deregistration (Unregister)
      • List all buckets (List All)
      • List all bucket detailed info (List All Info)
      • Count all buckets (Count All)
      • Delete CSP bucket directly (Delete CSP)
      • ※ Available only via CB-Spider standard REST API (Basic Auth / JSON)

• CSP Support Status

  CSP	Bucket	Object	Multipart Upload	Versioning	CORS	Special
  AWS	O	O	O	O	O	O
  AZURE	O	O	NA	NA	NA	O
  GCP	O	O	O	O	O	O
  ALIBABA (Excl. Mainland China)	O	O	O	O	O	O
  TENCENT	O	O	O	O	O	O
  IBM	O	O	O	O	O	O
  OPENSTACK	O	O	NA	NA	O	O
  NCP	O	O	O	NA	NA	O
  NHN	O	O	O	NA	NA	O
  KT	O	O	O	O	O	O

  ● Legend: O = Provided (Supported), NA = Not Applicable (CSP does not support or unstable)

  • ※ CSP-specific S3 Notes: See PR #1618

    • Deleted bucket names cannot be reused for a certain period: AWS 48H-72H, Alibaba 4H-8H, etc.

    • Alibaba: Mainland China regions excluded (additional settings such as Domain configuration required)

    • Azure: Multipart Upload / Versioning / CORS are not supported due to Azure structural constraints (details)

    • Azure: Azure-specific headers required when calling Upload REST API (reference)

  • ※ Error Message Examples for Unsupported Features

3. CB-Spider S3 Configuration

• S3 Pre-check: Verify that Object Storage is enabled and permissions are configured in the CSP Console
• S3 Credentials: Obtain S3 credentials (HMAC key) for each CSP (CSP-specific Guide)
• S3 Credential Registration: S3 credentials must be added to CB-Spider Connection's Credential information

  • CSPs where existing Credentials are identical and no additional S3 credentials are needed

    [AWS/Alibaba/Tencent/NCP]
    

  • CSPs where S3 credentials need to be added to existing Credentials

    [GCP/Azure/IBM/OpenStack/NHN/KT]
    

    • (1) Using Key Names as provided by each CSP Web Console

      [GCP]
        curl -X POST "http://localhost:1024/spider/credential" \
          -H 'Content-Type: application/json' \
          -d '{
            "CredentialName": "gcp-credential01",
            "ProviderName": "GCP",
            "KeyValueInfoList": [
              {
                "Key": "PrivateKey",
                "Value": "-----BEGIN PRIVATE KEY-----\nxxx\n-----END PRIVATE KEY-----\n"
              },
              { "Key": "ProjectID",         "Value": "project" },
              { "Key": "ClientEmail",       "Value": "user@user.iam.gserviceaccount.com" },
      
              { "Key": "Access Key",       "Value": "XXXX" },   <======= for S3
              { "Key": "Secret",       "Value": "YYYY" }        <======= for S3
            ]
          }'
      
      [Azure]
          ... existing Credential information
      
          {"Key": "StorageAccountName", "Value": "***"},
          {"Key": "StorageAccountKey", "Value": "***"}
      
      [IBM]
          ... existing Credential information
      
          {"Key": "access_key_id", "Value": "***"},
          {"Key": "secret_access_key", "Value": "***"}
      
      [OpenStack]
          ... existing Credential information
      
          {"Key": "access", "Value": "***"},
          {"Key": "secret", "Value": "***"}
      
      [NHN/KT]
          ... existing Credential information ...
      
          {"Key": "Access Key", "Value": "***"},
          {"Key": "Secret Key", "Value": "***"}
      

    • (2) Using unified abstract Key Names

      [GCP/IBM/OpenStack/NHN/KT]
          ... existing Credential information ...
      
          {"Key": "S3AccessKey", "Value": "***"},
          {"Key": "S3SecretKey", "Value": "***"}
      

4. CB-Spider S3 API Usage

• CB-Spider Object Storage API Guide (CB Standard, Basic Auth / JSON Format)
• CB-Spider Object Storage API Guide (AWS Compatible, Basic Auth / XML Format)
• CB-Spider Object Storage API Guide (AWS Compatible, SigV4 / XML Format)
• CB-Spider Object Storage API Swagger UI

5. CB-Spider AdminWeb S3 Usage

• Select a Connection
• Navigate to the S3 management page
• Key features of S3 management are as follows:
  • (1) Create a Bucket
  • (2) View and select created Bucket information
  • (3) Store Objects in the selected Bucket
  • (4) View stored Object information
  • (5) Download stored Objects
  • (6) Generate PreSigned URLs for stored Objects
  • (7) Toggle Versioning for the current Bucket's Objects
  • (8) Manage versions of Objects in Versioning-enabled Buckets
  • (9) Configure and delete CORS settings for the current Bucket

6. S3 Browser S3 Explorer Tool Usage

• For installation and usage, refer to S3 Browser
• CB-Spider Account configuration:
  • Account: Select S3 Compatible Storage
  • API: Enter localhost:1024/spider/s3
  • Access Key: Enter SPIDER_USERNAME@ConnectionName
  • Secret Access: Enter SPIDER_PASSWORD
  • Use secure transfer (SSL/TLS): unCheck

7. rclone Multi-Cloud Storage Sync/Copy Tool Usage

• For installation and usage, refer to rclone
• CB-Spider Account configuration:
  • vi ~/.config/rclone/rclone.conf

    [mydrv]
    type = s3
    provider = Other
    access_key_id = `SPIDER_USERNAME@ConnectionName`
    secret_access_key = `SPIDER_PASSWORD`
    endpoint = http://localhost:1024/spider/s3
    acl = public
    location_constraint = ap-southeast-2
    

• Example commands:

  $ rclone config show
  [mydrv]
  type = s3
  provider = Other
  access_key_id = admin@aws-config01
  secret_access_key = ****
  endpoint = http://localhost:1024/spider/s3
  acl = private
  location_constraint = ap-southeast-2
  
  $ rclone mkdir mydrv:test-bucket-powerkim-001
  $ rclone lsd mydrv:
          -1 2026-07-13 10:46:18        -1 test-bucket-powerkim-001
  $ rclone copy 1.txt mydrv:test-bucket-powerkim-001
  $ rclone ls mydrv:test-bucket-powerkim-001
        2 1.txt
  $ rclone purge mydrv:test-bucket-powerkim-001
  $ rclone lsd mydrv:
  

---