Aws::Client::ClientConfiguration default construction is very slow #2398

Gribouillages · 2023-03-21T14:58:29Z

Gribouillages
Mar 21, 2023

When I use Aws::Client::ClientConfiguration for default construction, it takes several seconds. I checked the trace log and found that it sends some http requests when constructing. Can I close these requests or make it construct quickly?

#include <aws/s3/S3Client.h>
#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentialsProvider.h>
#include <iostream>
#include <ctime>

int main()
{
    time_t start, end;

    Aws::SDKOptions options;
    options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
    Aws::InitAPI(options);

    start = clock();
    Aws::Client::ClientConfiguration cfg; //4 ~ 6s
    end = clock();

    cfg.endpointOverride = "xx";
    cfg.scheme = Aws::Http::Scheme::HTTP;
    cfg.verifySSL = false;
    cfg.region = "xx";


    Aws::Auth::AWSCredentials cred("xxxx", "xxxx");

    auto client = std::make_shared<Aws::S3::S3Client>(cred, cfg,
                                                     Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, false);

    std::cout << (end-start) << std::endl;
    Aws::ShutdownAPI(options);
}

Answered by jmklix

Mar 21, 2023

Can you make sure you are using the sdk correctly by adding the sdk call in curly brackets:

#include <aws/core/Aws.h>
int main(int argc, char** argv)
{
   Aws::SDKOptions options;
   Aws::InitAPI(options);
   {
      // make your SDK calls here.
   }
   Aws::ShutdownAPI(options);
   return 0;
}

View full answer

jmklix · 2023-03-21T18:12:15Z

jmklix
Mar 21, 2023
Maintainer

Can you make sure you are using the sdk correctly by adding the sdk call in curly brackets:

#include <aws/core/Aws.h>
int main(int argc, char** argv)
{
   Aws::SDKOptions options;
   Aws::InitAPI(options);
   {
      // make your SDK calls here.
   }
   Aws::ShutdownAPI(options);
   return 0;
}

4 replies

Gribouillages Mar 22, 2023
Author

I modified the code, but the same problem still exists.

#include <aws/s3/S3Client.h>
#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentialsProvider.h>
#include <iostream>
#include <ctime>

int main()
{
    time_t start, end;

    Aws::SDKOptions options;
    options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
    Aws::InitAPI(options);
    {
        start = clock();
        Aws::Client::ClientConfiguration cfg; //4 ~ 6s
        end = clock();

        cfg.endpointOverride = "xx";
        cfg.scheme = Aws::Http::Scheme::HTTP;
        cfg.verifySSL = false;
        cfg.region = "xx";


        Aws::Auth::AWSCredentials cred("xxxx", "xxxx");

        auto client = std::make_shared<Aws::S3::S3Client>(cred, cfg,
                                                          Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, false);

        std::cout << (end-start) << std::endl;
    }
    Aws::ShutdownAPI(options);
    return 0;
}

here are some trace logs generated during construction.

[DEBUG] 2023-03-22 02:15:20.549 HttpClientFactory [140566085967808] Initializing Http Static State
[DEBUG] 2023-03-22 02:15:20.549 HttpClientFactory [140566085967808] Initializing Curl Http Client
[INFO] 2023-03-22 02:15:20.549 CurlHttpClient [140566085967808] Initializing Curl library with version: 7.58.0, ssl version: OpenSSL/1.1.1
[INFO] 2023-03-22 02:15:20.550 EC2MetadataClient [140566085967808] Using IMDS endpoint: http://169.254.169.254
[DEBUG] 2023-03-22 02:15:20.550 ClientConfiguration [140566085967808] ClientConfiguration will use SDK Auto Resolved profile: [default] if not specified by users.
[WARN] 2023-03-22 02:15:20.550 ClientConfiguration [140566085967808] Retry Strategy will use the default max attempts.
[INFO] 2023-03-22 02:15:20.550 EC2MetadataClient [140566085967808] Creating AWSHttpResourceClient with max connections 2 and scheme http
[INFO] 2023-03-22 02:15:20.550 CurlHandleContainer [140566085967808] Initializing CurlHandleContainer with size 2
[DEBUG] 2023-03-22 02:15:20.550 ClientConfiguration [140566085967808] ClientConfiguration will use SDK Auto Resolved profile: [default] if not specified by users.
[WARN] 2023-03-22 02:15:20.550 ClientConfiguration [140566085967808] Retry Strategy will use the default max attempts.
[TRACE] 2023-03-22 02:15:20.550 EC2MetadataClient [140566085967808] Getting current region for ec2 instance
[TRACE] 2023-03-22 02:15:20.550 EC2MetadataClient [140566085967808] Calling EC2MetadataService to get token
[TRACE] 2023-03-22 02:15:20.550 EC2MetadataClient [140566085967808] Retrieving credentials from http://169.254.169.254/latest/api/token
[TRACE] 2023-03-22 02:15:20.550 CurlHttpClient [140566085967808] Making request to http://169.254.169.254/latest/api/token
[TRACE] 2023-03-22 02:15:20.550 CurlHttpClient [140566085967808] Including headers:
[TRACE] 2023-03-22 02:15:20.550 CurlHttpClient [140566085967808] host: 169.254.169.254
[TRACE] 2023-03-22 02:15:20.550 CurlHttpClient [140566085967808] user-agent: aws-sdk-cpp/1.11.10 Linux/5.4.0-144-generic x86_64 GCC/7.5.0
[TRACE] 2023-03-22 02:15:20.550 CurlHttpClient [140566085967808] x-aws-ec2-metadata-token-ttl-seconds: 21600
[DEBUG] 2023-03-22 02:15:20.550 CurlHandleContainer [140566085967808] Attempting to acquire curl connection.
[DEBUG] 2023-03-22 02:15:20.550 CurlHandleContainer [140566085967808] No current connections available in pool. Attempting to create new connections.
[DEBUG] 2023-03-22 02:15:20.550 CurlHandleContainer [140566085967808] attempting to grow pool size by 2
[INFO] 2023-03-22 02:15:20.550 CurlHandleContainer [140566085967808] Pool grown by 2
[INFO] 2023-03-22 02:15:20.550 CurlHandleContainer [140566085967808] Connection has been released. Continuing.
[DEBUG] 2023-03-22 02:15:20.550 CurlHandleContainer [140566085967808] Returning connection handle 0x55e82b69ea00
[DEBUG] 2023-03-22 02:15:20.550 CurlHttpClient [140566085967808] Obtained connection handle 0x55e82b69ea00
[DEBUG] 2023-03-22 02:15:20.550 CURL [140566085967808] (Text)   Trying 169.254.169.254...

[DEBUG] 2023-03-22 02:15:20.550 CURL [140566085967808] (Text) TCP_NODELAY set

[DEBUG] 2023-03-22 02:15:20.556 CURL [140566085967808] (Text) Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)

[DEBUG] 2023-03-22 02:15:20.556 CURL [140566085967808] (HeaderOut) PUT /latest/api/token HTTP/1.1
host: 169.254.169.254
Accept: */*
Connection: Upgrade, HTTP2-Settings
Upgrade: h2c
HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
user-agent: aws-sdk-cpp/1.11.10 Linux/5.4.0-144-generic x86_64 GCC/7.5.0
x-aws-ec2-metadata-token-ttl-seconds: 21600


[DEBUG] 2023-03-22 02:15:22.552 CURL [140566085967808] (Text) Operation too slow. Less than 1 bytes/sec transferred the last 1 seconds

[DEBUG] 2023-03-22 02:15:22.552 CURL [140566085967808] (Text) stopped the pause stream!

[DEBUG] 2023-03-22 02:15:22.552 CURL [140566085967808] (Text) Closing connection 0

[ERROR] 2023-03-22 02:15:22.552 CurlHttpClient [140566085967808] Curl returned error code 28 - Timeout was reached
[DEBUG] 2023-03-22 02:15:22.552 CurlHandleContainer [140566085967808] Destroy curl handle: 0x55e82b69ea00
[DEBUG] 2023-03-22 02:15:22.552 CurlHandleContainer [140566085967808] Created replacement handle and released to pool: 0x55e82b69ea00
[ERROR] 2023-03-22 02:15:22.552 EC2MetadataClient [140566085967808] Http request to retrieve credentials failed
[WARN] 2023-03-22 02:15:22.552 EC2MetadataClient [140566085967808] Request failed, now waiting 0 ms before attempting again.
[TRACE] 2023-03-22 02:15:22.552 CurlHttpClient [140566085967808] Making request to http://169.254.169.254/latest/api/token
[TRACE] 2023-03-22 02:15:22.552 CurlHttpClient [140566085967808] Including headers:
[TRACE] 2023-03-22 02:15:22.552 CurlHttpClient [140566085967808] host: 169.254.169.254
[TRACE] 2023-03-22 02:15:22.552 CurlHttpClient [140566085967808] user-agent: aws-sdk-cpp/1.11.10 Linux/5.4.0-144-generic x86_64 GCC/7.5.0
[TRACE] 2023-03-22 02:15:22.552 CurlHttpClient [140566085967808] x-aws-ec2-metadata-token-ttl-seconds: 21600
[DEBUG] 2023-03-22 02:15:22.552 CurlHandleContainer [140566085967808] Attempting to acquire curl connection.
[INFO] 2023-03-22 02:15:22.552 CurlHandleContainer [140566085967808] Connection has been released. Continuing.
[DEBUG] 2023-03-22 02:15:22.552 CurlHandleContainer [140566085967808] Returning connection handle 0x55e82b69ea00
[DEBUG] 2023-03-22 02:15:22.552 CurlHttpClient [140566085967808] Obtained connection handle 0x55e82b69ea00
[DEBUG] 2023-03-22 02:15:22.552 CURL [140566085967808] (Text)   Trying 169.254.169.254...

[DEBUG] 2023-03-22 02:15:22.552 CURL [140566085967808] (Text) TCP_NODELAY set
[DEBUG] 2023-03-22 02:15:22.552 CURL [140566085967808] (Text) Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)

[DEBUG] 2023-03-22 02:15:22.552 CURL [140566085967808] (HeaderOut) PUT /latest/api/token HTTP/1.1
host: 169.254.169.254
Accept: */*
Connection: Upgrade, HTTP2-Settings
Upgrade: h2c
HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
user-agent: aws-sdk-cpp/1.11.10 Linux/5.4.0-144-generic x86_64 GCC/7.5.0
x-aws-ec2-metadata-token-ttl-seconds: 21600


[DEBUG] 2023-03-22 02:15:23.553 CURL [140566085967808] (Text) Operation too slow. Less than 1 bytes/sec transferred the last 1 seconds

[DEBUG] 2023-03-22 02:15:23.553 CURL [140566085967808] (Text) stopped the pause stream!

[DEBUG] 2023-03-22 02:15:23.553 CURL [140566085967808] (Text) Closing connection 0

[ERROR] 2023-03-22 02:15:23.553 CurlHttpClient [140566085967808] Curl returned error code 28 - Timeout was reached
[DEBUG] 2023-03-22 02:15:23.553 CurlHandleContainer [140566085967808] Destroy curl handle: 0x55e82b69ea00
[DEBUG] 2023-03-22 02:15:23.553 CurlHandleContainer [140566085967808] Created replacement handle and released to pool: 0x55e82b69ea00
[ERROR] 2023-03-22 02:15:23.553 EC2MetadataClient [140566085967808] Http request to retrieve credentials failed
[ERROR] 2023-03-22 02:15:23.553 EC2MetadataClient [140566085967808] Can not retrieve resource from http://169.254.169.254/latest/api/token
[TRACE] 2023-03-22 02:15:23.553 EC2MetadataClient [140566085967808] Calling EC2MetadataService to get token failed, falling back to less secure way.
[TRACE] 2023-03-22 02:15:23.553 EC2MetadataClient [140566085967808] Getting default credentials for ec2 instance from http://169.254.169.254
[TRACE] 2023-03-22 02:15:23.553 EC2MetadataClient [140566085967808] Retrieving credentials from http://169.254.169.254/latest/meta-data/iam/security-credentials
[TRACE] 2023-03-22 02:15:23.553 CurlHttpClient [140566085967808] Making request to http://169.254.169.254/latest/meta-data/iam/security-credentials
[TRACE] 2023-03-22 02:15:23.553 CurlHttpClient [140566085967808] Including headers:
[TRACE] 2023-03-22 02:15:23.553 CurlHttpClient [140566085967808] host: 169.254.169.254
[TRACE] 2023-03-22 02:15:23.553 CurlHttpClient [140566085967808] user-agent: aws-sdk-cpp/1.11.10 Linux/5.4.0-144-generic x86_64 GCC/7.5.0
[DEBUG] 2023-03-22 02:15:23.553 CurlHandleContainer [140566085967808] Attempting to acquire curl connection.
[INFO] 2023-03-22 02:15:23.553 CurlHandleContainer [140566085967808] Connection has been released. Continuing.
[DEBUG] 2023-03-22 02:15:23.553 CurlHandleContainer [140566085967808] Returning connection handle 0x55e82b69ea00
[DEBUG] 2023-03-22 02:15:23.553 CurlHttpClient [140566085967808] Obtained connection handle 0x55e82b69ea00
[DEBUG] 2023-03-22 02:15:23.553 CURL [140566085967808] (Text)   Trying 169.254.169.254...

[DEBUG] 2023-03-22 02:15:23.553 CURL [140566085967808] (Text) TCP_NODELAY set

[DEBUG] 2023-03-22 02:15:23.553 CURL [140566085967808] (Text) Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)

[DEBUG] 2023-03-22 02:15:23.553 CURL [140566085967808] (HeaderOut) GET /latest/meta-data/iam/security-credentials HTTP/1.1
host: 169.254.169.254
Accept: */*
Connection: Upgrade, HTTP2-Settings
Upgrade: h2c
HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
user-agent: aws-sdk-cpp/1.11.10 Linux/5.4.0-144-generic x86_64 GCC/7.5.0


[DEBUG] 2023-03-22 02:15:24.554 CURL [140566085967808] (Text) Operation too slow. Less than 1 bytes/sec transferred the last 1 seconds

[DEBUG] 2023-03-22 02:15:24.554 CURL [140566085967808] (Text) stopped the pause stream!

[DEBUG] 2023-03-22 02:15:24.554 CURL [140566085967808] (Text) Closing connection 0

[ERROR] 2023-03-22 02:15:24.554 CurlHttpClient [140566085967808] Curl returned error code 28 - Timeout was reached
[DEBUG] 2023-03-22 02:15:24.554 CurlHandleContainer [140566085967808] Destroy curl handle: 0x55e82b69ea00
[DEBUG] 2023-03-22 02:15:24.554 CurlHandleContainer [140566085967808] Created replacement handle and released to pool: 0x55e82b6a7f90
[ERROR] 2023-03-22 02:15:24.554 EC2MetadataClient [140566085967808] Http request to retrieve credentials failed
[WARN] 2023-03-22 02:15:24.554 EC2MetadataClient [140566085967808] Request failed, now waiting 0 ms before attempting again.
[TRACE] 2023-03-22 02:15:24.554 CurlHttpClient [140566085967808] Making request to http://169.254.169.254/latest/meta-data/iam/security-credentials
.......

jmklix Mar 22, 2023
Maintainer

I don't think you can close those requests. Also when reproducing this I'm only seeing 29ms from start to end. What OS and device are you running this on?

Gribouillages Mar 23, 2023
Author

my OS is Ubuntu18.04, and the version of the SDK is 1.11.10. I have skipped these requests by setting the environment variable AWS_EC2_METADATA_DISABLED=true, which enables me to quickly construct.

Jon118Lexi Aug 8, 2024

if you don't want to set environment, you can use this
Aws::Client::ClientConfiguration cfg("default", true);
it's the same result.

2023-12-01T20:01:00Z

github-actions[bot]
Bot Dec 1, 2023

Hello! Reopening this discussion to make it searchable.

0 replies

Navigation Menu

Uh oh!

Aws::Client::ClientConfiguration default construction is very slow #2398

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Aws::Client::ClientConfiguration default construction is very slow #2398

Uh oh!

Gribouillages Mar 21, 2023

Replies: 2 comments · 4 replies

Uh oh!

jmklix Mar 21, 2023 Maintainer

Uh oh!

Gribouillages Mar 22, 2023 Author

Uh oh!

jmklix Mar 22, 2023 Maintainer

Uh oh!

Gribouillages Mar 23, 2023 Author

Uh oh!

Jon118Lexi Aug 8, 2024

Uh oh!

github-actions[bot] Bot Dec 1, 2023

Gribouillages
Mar 21, 2023

Replies: 2 comments 4 replies

jmklix
Mar 21, 2023
Maintainer

Gribouillages Mar 22, 2023
Author

jmklix Mar 22, 2023
Maintainer

Gribouillages Mar 23, 2023
Author

github-actions[bot]
Bot Dec 1, 2023