Skip to content

Fix various problems with S3 related code - #3090

Merged
WardF merged 3 commits into
Unidata:mainfrom
DennisHeimbigner:s3apiupdate.dmh
Apr 14, 2025
Merged

Fix various problems with S3 related code#3090
WardF merged 3 commits into
Unidata:mainfrom
DennisHeimbigner:s3apiupdate.dmh

Conversation

@DennisHeimbigner

Copy link
Copy Markdown
Contributor

re: PR #3068

Part of splitting PR 3068.

General goal is to fix minor bugs and issues involving S3 code outside of libnczarr.

  1. libsrc/s3io.c:
    • Fix handling of error output of NC_s3sdkinfo.
    • Implement delete arg to s3io_close
  2. Add support for Zarr-Over-HTTP (ZOH) protocol, but leave disabled until given the go-ahead from Manuel Reis.
  3. Change the ncs3sdk API to light of eventual addition of Zarr V3 support.
  4. Modify s3cleanup.in to catch and remove some previously overlooked entries in the Unidata S3 test bucket.
  5. Modify s3gc.in to catch and properly remove Unidata S3 test bucket entries with old UIDs.
  6. Modify ds3util.c:
    • Support use of a non-standard port -- H/T Manuel Reis
    • Clean up some memory leaks
    • Add disabled ZOH support
    • Better doc on how URLs are interpreted.
    • Document how .aws/config and .aws/credentials files are parsed.
  7. Repair bugs in the internal S3 reader/writer module.
    • Allow API functions to return an http code (see also nch5s3comms.h).
    • Fix some casting warnings.
  8. Fixes to ncs3sdk_h5.c, the dispatch wrapper for the internal S3 reader/writer
    • Move httptonc to this file.
    • Make conform to API modifications
    • Refactor the key search code.
    • Rename getkeys to list and searchkeys to listall.
  9. Fixes to nczarr_test/s3util.c
    • Reflect ds3util.c API changes: e.g search->listall.
  10. Add s3util.c to v3_nczarr_directory
  11. Extend unit_test/test_s3dk.c to reflect S3 API changes.
  12. Extend unit_test/run_s3sdk.sh to add new tests involving S3 API changes.

re: PR Unidata#3068

Part of splitting PR 3068.

General goal is to fix minor bugs and issues involving
S3 code outside of libnczarr.

1. libsrc/s3io.c:
    - Fix handling of error output of NC_s3sdkinfo.
    - Implement delete arg to s3io_close
2. Add support for Zarr-Over-HTTP (ZOH) protocol, but leave disabled until given the go-ahead from Manuel Reis.
3. Change the ncs3sdk API to light of eventual addition of Zarr V3 support.
4. Modify s3cleanup.in to catch and remove some previously overlooked entries in the Unidata S3 test bucket.
5. Modify s3gc.in to catch and properly remove Unidata S3 test bucket entries with old UIDs.
6. Modify ds3util.c:
    - Support use of a non-standard port -- H/T Manuel Reis
    - Clean up some memory leaks
    - Add disabled ZOH support
    - Better doc on how URLs are interpreted.
    - Document how .aws/config and .aws/credentials files are parsed.
7. Repair bugs in the internal S3 reader/writer module.
    - Allow API functions to return an http code (see also nch5s3comms.h).
    - Fix some casting warnings.
8. Fixes to ncs3sdk_h5.c, the dispatch wrapper for the internal S3 reader/writer
    - Move httptonc to this file.
    - Make conform to API modifications
    - Refactor the key search code.
    - Rename getkeys to list and searchkeys to listall.
9. Fixes to nczarr_test/s3util.c
    - Reflect ds3util.c API changes: e.g search->listall.
10. Add s3util.c to v3_nczarr_directory
11. Extend unit_test/test_s3dk.c to reflect S3 API changes.
12. Extend unit_test/run_s3sdk.sh to add new tests involving S3 API changes.
@WardF
WardF merged commit 417e169 into Unidata:main Apr 14, 2025
Comment thread libdispatch/nch5s3comms.c
#include "netcdf.h"
#include "ncuri.h"
#include "ncutil.h"
#include "netcdf_vutils.h"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to cause by default builds config to fail! Since there's not such header.

mannreis added a commit to mannreis/netcdf-c that referenced this pull request Jun 16, 2025
…date.dmh"

This reverts commit 417e169, reversing
changes made to 90fc96e.
@DennisHeimbigner
DennisHeimbigner deleted the s3apiupdate.dmh branch July 12, 2025 02:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants