Store Gateway performance issue when migrate storage from CEPH to VAST.

[chailuecha]

Hello,

I am having issue with my storage backend migration. Would like to get some advice.
My migration from CEPH to VAST. steps.

Use custom tool based on HTTP Rest s3 API to copy bucket from CEPH to VAST

2026-01-29T02:09:42Z INFO  data_migration] Successfully put key: anonymous/phlaredb/01KF2X3000GK1ECK4349YQETSZ/symbols/stacktraces.symdb

point storage to Vast endpoint. This automatically work since both are s3 compatible.

System can startup but somehow the query is much slower. When connecting to CEPH. Merge profile is super fast. Even the case that I can see some parquet files in bucket are as big as 20GB.

method=POST uri=/storegateway.v1.StoreGatewayService/MergeProfilesStacktraces status=200 duration=558.709974ms 

When I connect to VAST the merge tooks 40+ secs.
I already test downloading form both system. It took approximately the same amount of time. So I don't think it is about storage implementation.

I believe store gateway periodically sync blocks to disk in advance. So the query should just read from local disk and merge profile. Would like to ask if anything might cause this problem? or feel free to point out how should I debug this.

Thanks a ton

[chailuecha]

Would like to close this. I eventually found it is about s3 credential overhead in bucket store

[chailuecha]

It is specific to my system that doesn't need acesskey for credential and it trigger objectStore to try to resolve accessKey by calling external endpoint.