Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50760

Loading travel sample bucket failed due to the loader exited - 'failed to execute query: ambiguous timeout'

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • 7.1.0
    • tools
    • Enterprise Edition 7.1.0 build 2218

    Description

       

      Steps:

      • 111 node cluster with multiple services
      • Create couchbase bucket and start data-loading
      • Load travel-sample bucket in parallel to the prev data load

      Observation:

      'travel-sample' bucket created successfully but loader crashed with the following reason,

      (404) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'

      Loading sample bucket travel-sample failed. Samples loader exited with status 1.
      		 
      Loader's output was:
      		 
       
      		 
      2022-02-02T10:51:21.942-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools'
      		 
      2022-02-02T10:51:21.945-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.122.100:8091/pools'
      		 
      2022-02-02T10:51:21.945-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools/default'
      		 
      2022-02-02T10:51:22.275-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.122.100:8091/pools/default'
      		 
      2022-02-02T10:51:22.289-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets'
      		 
      2022-02-02T10:51:22.434-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets'
      		 
      2022-02-02T10:51:22.435-08:00 (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"d17e3dbbb234b1f0ca3fb137a960215f","developer_preview":false,"version":{"min_version":"7.1.0","is_mixed_cluster":false},"max_vbuckets":1024,"uniform_vbuckets":true}
      		 
      2022-02-02T10:51:22.568-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:22.570-08:00 (REST) (Attempt 1) (GET) (404) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:22.570-08:00 WARN: (REST) (Attempt 1) (GET) Request to endpoint '/pools/default/buckets/travel-sample' failed with status code 404 -- logging.(*ToolsCommonLogger).Log() at tools_common.go:28
      		 
      2022-02-02T10:51:22.570-08:00 (Plan) (Bucket) Creating remote bucket 'travel-sample' if it does not exist
      		 
      2022-02-02T10:51:22.570-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:22.571-08:00 (REST) (Attempt 1) (GET) (404) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:22.571-08:00 WARN: (REST) (Attempt 1) (GET) Request to endpoint '/pools/default/buckets/travel-sample' failed with status code 404 -- logging.(*ToolsCommonLogger).Log() at tools_common.go:28
      		 
      2022-02-02T10:51:22.571-08:00 Bucket travel-sample does not exist, creating one with the specified settings
      		 
      2022-02-02T10:51:22.572-08:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets'
      		 
      2022-02-02T10:51:22.716-08:00 (REST) (Attempt 1) (POST) (202) Received response from 'http://172.23.122.100:8091/pools/default/buckets'
      		 
      2022-02-02T10:51:22.716-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:23.107-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:23.717-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:23.966-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:24.717-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:24.854-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:25.717-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:25.848-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:26.717-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:26.829-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:26.829-08:00 (Plan) (Bucket) Successfully ensured bucket 'travel-sample' exists | {"number":1,"duration":"4.259338105s"}
      		 
      2022-02-02T10:51:26.829-08:00 (Plan) (Bucket) Automatically creating/editing scopes/collections for bucket 'travel-sample'
      		 
      2022-02-02T10:51:26.830-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:26.948-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:26.949-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample/scopes'
      		 
      2022-02-02T10:51:26.950-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample/scopes'
      		 
      2022-02-02T10:51:26.950-08:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample/scopes/@ensureManifest/0'
      		 
      2022-02-02T10:51:26.953-08:00 (REST) (Attempt 1) (POST) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample/scopes/@ensureManifest/0'
      		 
      2022-02-02T10:51:26.954-08:00 (REST) (Attempt 1) (PUT) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample/scopes?validOnUid=0'
      		 
      2022-02-02T10:51:27.053-08:00 (REST) (Attempt 1) (PUT) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample/scopes?validOnUid=0'
      		 
      2022-02-02T10:51:27.053-08:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample/scopes/@ensureManifest/1'
      		 
      2022-02-02T10:51:27.416-08:00 (REST) (Attempt 1) (POST) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample/scopes/@ensureManifest/1'
      		 
      2022-02-02T10:51:27.416-08:00 (Plan) (Bucket) Successfully created/edited scopes/collections for bucket 'travel-sample' | {"number":2,"duration":"586.730805ms"}
      		 
      2022-02-02T10:51:27.416-08:00 (Plan) (Bucket) Transferring bucket configuration for bucket 'travel-sample'
      		 
      2022-02-02T10:51:27.417-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:27.936-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:27.937-08:00 Editing bucket to conform with settings provided
      		 
      2022-02-02T10:51:27.937-08:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:28.144-08:00 (REST) (Attempt 1) (POST) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:28.144-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:28.515-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.122.100:8091/pools/default/buckets/travel-sample'
      		 
      2022-02-02T10:51:28.516-08:00 (Plan) (Bucket) Successfully transferred bucket configuration for bucket 'travel-sample' | {"number":3,"duration":"1.09975535s"}
      		 
      2022-02-02T10:51:28.516-08:00 (Plan) (Index) Transferring views definitions for bucket 'travel-sample'
      		 
      2022-02-02T10:51:28.641-08:00 (Plan) (Index) Successfully transferred views definitions for bucket 'travel-sample' | {"number":4,"duration":"125.257829ms"}
      		 
      2022-02-02T10:51:28.641-08:00 (Plan) (Query) Transferring Query metadata for bucket 'travel-sample'
      		 
      2022-02-02T10:51:28.641-08:00 (Plan) (Query) Successfully transferred Query metadata for bucket 'travel-sample' | {"number":5,"duration":"21.324µs"}
      		 
      2022-02-02T10:51:28.641-08:00 (Plan) (Query) Executing queries for bucket 'travel-sample'
      		 
      2022-02-02T10:51:28.740-08:00 (Couchbase) Creating Query connection to cluster 'couchbase://172.23.122.100:11210,172.23.122.101:11210,172.23.122.102:11210,172.23.122.159:11210,172.23.122.160:11210,172.23.122.163:11210,172.23.122.165:11210,172.23.122.166:11210,172.23.122.167:11210,172.23.122.168:11210,172.23.122.169:11210,172.23.122.170:11210,172.23.122.171:11210,172.23.122.172:11210,172.23.122.173:11210,172.23.122.174:11210,172.23.122.175:11210,172.23.122.176:11210'
      		 
      2022-02-02T10:51:28.911-08:00 (Couchbase) Successfully created Query connection to cluster 'couchbase://172.23.122.100:11210,172.23.122.101:11210,172.23.122.102:11210,172.23.122.159:11210,172.23.122.160:11210,172.23.122.163:11210,172.23.122.165:11210,172.23.122.166:11210,172.23.122.167:11210,172.23.122.168:11210,172.23.122.169:11210,172.23.122.170:11210,172.23.122.171:11210,172.23.122.172:11210,172.23.122.173:11210,172.23.122.174:11210,172.23.122.175:11210,172.23.122.176:11210'
      		 
      2022-02-02T10:51:58.915-08:00 JSON import failed: failed to execute cluster operations: failed to execute bucket operation for bucket 'travel-sample': failed to execute N1QL queries for bucket 'travel-sample': failed to execute queries against sink bucket: failed to execute query: ambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"ambiguous timeout"}},"OperationID":"http","Opaque":"4e0b0173-d094-423a-876d-7c53b1382819","TimeObserved":30002356394,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"http://172.23.122.120:8093","LastDispatchedFrom":"","LastConnectionID":""} | {"statement":"CREATE PRIMARY INDEX def_primary on `travel-sample` USING GSI WITH {\"defer_build\":true}","client_context_id":"4e0b0173-d094-423a-876d-7c53b1382819"}
      		 
      JSON import failed: operation has timed out
      		 
      hide

       

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty