Details
-
Bug
-
Resolution: Fixed
-
Major
-
6.6.0
-
Untriaged
-
1
-
No
Description
What's the issue?
cbbackupmgr can take a long time to return to the user stating that they haven't provided sufficient credentials.
Why is it happening?
When we turn on the SDK request with retries logging we can see that the SDK is trying to fetch the ec2metadata (which happens irrelevant of being in ec2). We can see that it tries this request a handful of times with an extended timeout for each. This results in a 5-10 minute delay before the user is informed that they haven't provided sufficient credentials.
Steps to reproduce
Run a config with "AWS_SDK_LOAD_CONFIG" set to false and "CB_AWS_FORCE_ENABLE_LOGGING" set to "debug-with-retries" and manually supply the "--obj-region" flag.
We should see the following:
2020/06/09 09:32:41 (AWS) Force enabled logging
|
2020/06/09 09:32:41 DEBUG: Request ec2metadata/GetToken Details:
|
---[ REQUEST POST-SIGN ]-----------------------------
|
PUT /latest/api/token HTTP/1.1
|
Host: 169.254.169.254
|
User-Agent: aws-sdk-go/1.27.1 (go1.14.4; linux; amd64)
|
Content-Length: 0
|
X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
|
Accept-Encoding: gzip
|
|
|
|
|
-----------------------------------------------------
|
2020/06/09 09:33:41 DEBUG: Send Request ec2metadata/GetToken failed, attempt 0/3, error RequestError: send request failed
|
caused by: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
2020/06/09 09:33:41 DEBUG: Request ec2metadata/GetToken Details:
|
---[ REQUEST POST-SIGN ]-----------------------------
|
PUT /latest/api/token HTTP/1.1
|
Host: 169.254.169.254
|
User-Agent: aws-sdk-go/1.27.1 (go1.14.4; linux; amd64)
|
Content-Length: 0
|
X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
|
Accept-Encoding: gzip
|
|
|
|
|
-----------------------------------------------------
|
2020/06/09 09:34:41 DEBUG: Send Request ec2metadata/GetToken failed, attempt 1/3, error RequestError: send request failed
|
caused by: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
2020/06/09 09:34:42 DEBUG: Request ec2metadata/GetToken Details:
|
---[ REQUEST POST-SIGN ]-----------------------------
|
PUT /latest/api/token HTTP/1.1
|
Host: 169.254.169.254
|
User-Agent: aws-sdk-go/1.27.1 (go1.14.4; linux; amd64)
|
Content-Length: 0
|
X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
|
Accept-Encoding: gzip
|
|
|
|
|
-----------------------------------------------------
|
2020/06/09 09:35:42 DEBUG: Send Request ec2metadata/GetToken failed, attempt 2/3, error RequestError: send request failed
|
caused by: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
2020/06/09 09:35:42 DEBUG: Request ec2metadata/GetToken Details:
|
---[ REQUEST POST-SIGN ]-----------------------------
|
PUT /latest/api/token HTTP/1.1
|
Host: 169.254.169.254
|
User-Agent: aws-sdk-go/1.27.1 (go1.14.4; linux; amd64)
|
Content-Length: 0
|
X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
|
Accept-Encoding: gzip
|
|
|
|
|
-----------------------------------------------------
|
2020/06/09 09:36:42 DEBUG: Send Request ec2metadata/GetToken failed, attempt 3/3, error RequestError: send request failed
|
caused by: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
2020/06/09 09:36:42 DEBUG: Request ec2metadata/GetMetadata Details:
|
---[ REQUEST POST-SIGN ]-----------------------------
|
GET /latest/meta-data/iam/security-credentials/ HTTP/1.1
|
Host: 169.254.169.254
|
User-Agent: aws-sdk-go/1.27.1 (go1.14.4; linux; amd64)
|
Accept-Encoding: gzip
|