With multi node cluster, upsert against IPv6 timesout

with gocbcore e5fbe974fa2c297b36bda375d742ba1419eff25f commit,

Upsert against Vulcan 5.1.0-1394 times out.

The cluster has 3 nodes with data, index, query, fts and eventing.

log

GOCB 11:30:56.903165 agent.go:348: SDK Version: gocb/v7.0.6
GOCB 11:30:56.903288 agent.go:349: Creating new agent: &{MemdAddrs:[[fd63:6f75:6368:2068:c8e3:a1ff:fed9:38ca]:11210] HttpAddrs:[[fd63:6f75:6368:2068:c8e3:a1ff:fed9:38ca]:8091] TlsConfig:<nil> BucketName:travel-sample Username:admin Password:password AuthHandler:<nil> UseMutationTokens:false UseKvErrorMaps:true UseEnhancedErrors:false HttpRedialPeriod:0s HttpRetryDelay:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s ServerConnectTimeout:7s NmvRetryDelay:100ms KvPoolSize:0 MaxQueueSize:0}
GOCB 11:30:56.903311 agent.go:496: Attempting to connect...
GOCB 11:30:56.903315 agent.go:499: Trying server at [fd63:6f75:6368:2068:c8e3:a1ff:fed9:38ca]:11210
GOCB 11:30:56.903319 agent.go:506: Trying to connect
GOCB 11:30:56.917010 agentrouting.go:51: Fetching cluster client data
GOCB 11:30:56.917062 memdclient.go:74: Writing request. [fd63:6f75:6368:1075:f964:44a2:2178:4cdd]:54836 OP=0x1f. Opaque=1
GOCB 11:30:56.927927 memdclient.go:188: Resolving response OP=0x1f. Opaque=1
GOCB 11:30:56.927940 memdclient.go:121: Dispatching response callback. OP=0x1f. Opaque=1
GOCB 11:30:56.927958 memdclient.go:74: Writing request. [fd63:6f75:6368:1075:f964:44a2:2178:4cdd]:54836 OP=0xfe. Opaque=2
GOCB 11:30:56.939580 memdclient.go:188: Resolving response OP=0xfe. Opaque=2
GOCB 11:30:56.939594 memdclient.go:121: Dispatching response callback. OP=0xfe. Opaque=2
GOCB 11:30:56.940274 agentrouting.go:85: Fetched error map: &{Version:1 Revision:1 Errors:map[205:{Name:SUBDOC_SUCCESS_DELETED Description:Subdoc: Success, but the affected document was (and still is) deleted Attributes:[item-deleted success subdoc] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 31:{Name:AUTH_STALE Description:Reauthentication required Attributes:[conn-state-invalidated auth] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 193:{Name:SUBDOC_PATH_MISMATCH Description:Subdoc: Path and document disagree on structure Attributes:[subdoc item-only] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 201:{Name:SUBDOC_PATH_EEXISTS Description:Subdoc: Path already exists Attributes:[subdoc item-only] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 194:{Name:SUBDOC_PATH_EINVAL Description:Subdoc: Invalid path (bad syntax or unacceptable semantics for command Attributes:[subdoc invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 1:{Name:KEY_ENOENT Description:Not Found Attributes:[item-only] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 209:{Name:SUBDOC_XATTR_UNKNOWN_VATTR Description:Subdoc: The server don't know about the specified virtual attribute Attributes:[subdoc invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 200:{Name:SUBDOC_DELTA_EINVAL Description:Subdoc: Invalid value passed for delta (out of range, or not an integer Attributes:[subdoc item-only] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 202:{Name:SUBDOC_VALUE_ETOODEEP Description:Subdoc: Value is too deep, or would make the document too deep Attributes:[subdoc invalid-input item-only] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 132:{Name:EINTERNAL Description:Internal error. Reconnect recommended Attributes:[internal conn-state-invalidated] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 6:{Name:DELTA_BADVAL Description:Existing document not a number Attributes:[item-only invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 0:{Name:SUCCESS Description:Success Attributes:[success] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 195:{Name:SUBDOC_PATH_E2BIG Description:Subdoc: Path size exceeds limit Attributes:[subdoc invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 35:{Name:ROLLBACK Description:Rollback Attributes:[dcp special-handling] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 3:{Name:E2BIG Description:Value is too big Attributes:[item-only invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 37:{Name:NOT_INITIALIZED Description:Server not initialized Attributes:[conn-state-invalidated] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 136:{Name:UNKNOWN_COLLECTION Description:Unknown collection (or no collection) in the key. Attributes:[invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 34:{Name:ERANGE Description:Invalid range requested Attributes:[invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 135:{Name:XATTR_EINVAL Description:Invalid extended attribute Attributes:[invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 204:{Name:SUBDOC_MULTI_PATH_FAILURE Description:Subdoc: Some (or all) commands failed. Inspect payload for details Attributes:[subdoc special-handling] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 212:{Name:SUBDOC_INVALID_XATTR_ORDER Description:Subdoc: Invalid XATTR order (xattrs should come first) Attributes:[subdoc invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 196:{Name:SUBDOC_PATH_E2DEEP Description:Subdoc: Path is too deep to be parsed Attributes:[subdoc invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 134:{Name:ETMPFAIL Description:Temporary failure. Try again Attributes:[temp retry-now] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 7:{Name:NOT_MY_VBUCKET Description:Server does not know about this vBucket Attributes:[fetch-config invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 9:{Name:LOCKED Description:Requested resource is locked Attributes:[item-locked item-only retry-now] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 32:{Name:AUTH_ERROR Description:Authentication failed Attributes:[conn-state-invalidated auth] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 198:{Name:SUBDOC_DOC_NOTJSON Description:Subdoc: Document not JSON Attributes:[subdoc item-only] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 211:{Name:SUBDOC_MULTI_PATH_FAILURE_DELETED Description:Subdoc: One or more paths in a multi-path command failed on a deleted document Attributes:[item-deleted subdoc special-handling] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 131:{Name:NOT_SUPPORTED Description:Command not supported with current bucket type/configuration Attributes:[support] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 197:{Name:SUBDOC_VALUE_CANTINSERT Description:Subdoc: Value invalid for insertion Attributes:[subdoc invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 33:{Name:AUTH_CONTINUE Description:Continue authentication processs Attributes:[special-handling] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 36:{Name:EACCESS Description:Not authorized for command Attributes:[support] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 4:{Name:EINVAL Description:Invalid packet Attributes:[internal invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 192:{Name:SUBDOC_PATH_ENOENT Description:Subdoc: Path not found in document Attributes:[subdoc item-only] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 5:{Name:NOT_STORED Description:Not Stored Attributes:[internal item-only] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 199:{Name:SUBDOC_NUM_ERANGE Description:Subdoc: Existing numeric value is not within range Attributes:[subdoc item-only] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 206:{Name:SUBDOC_XATTR_INVALID_FLAG_COMBO Description:Subdoc: The flag combination doesn't make any sense Attributes:[subdoc invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 203:{Name:SUBDOC_INVALID_COMBO Description:Subdoc: Lookup and mutation commands found within single packet Attributes:[subdoc invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 133:{Name:EBUSY Description:Busy, try again Attributes:[temp retry-now] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 2:{Name:KEY_EEXISTS Description:key already exists, or CAS mismatch Attributes:[item-only] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 130:{Name:ENOMEM Description:No memory available to store item. Add memory or remove some items and try later Attributes:[temp retry-later] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 208:{Name:SUBDOC_XATTR_UNKNOWN_MACRO Description:Subdoc: The server don't know about the specified macro Attributes:[subdoc invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 207:{Name:SUBDOC_XATTR_INVALID_KEY_COMBO Description:Subdoc: The key combination of the xattrs is not allowed Attributes:[subdoc invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 210:{Name:SUBDOC_XATTR_CANT_MODIFY_VATTR Description:Subdoc: Can't modify virtual attributes Attributes:[subdoc invalid-input] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 8:{Name:NO_BUCKET Description:Not connected to any bucket Attributes:[conn-state-invalidated] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}} 129:{Name:UNKNOWN_COMMAND Description:Unknown command. Maybe a newer server version knows about it Attributes:[support] Retry:{Strategy: Interval:0 After:0 Ceil:0 MaxDuration:0}}]}
GOCB 11:30:56.940293 agentrouting.go:110: Authenticating...
GOCB 11:30:56.940303 memdclient.go:74: Writing request. [fd63:6f75:6368:1075:f964:44a2:2178:4cdd]:54836 OP=0x21. Opaque=3
GOCB 11:30:56.951144 memdclient.go:188: Resolving response OP=0x21. Opaque=3
GOCB 11:30:56.951175 memdclient.go:121: Dispatching response callback. OP=0x21. Opaque=3
GOCB 11:30:56.951210 memdclient.go:74: Writing request. [fd63:6f75:6368:1075:f964:44a2:2178:4cdd]:54836 OP=0x89. Opaque=4
GOCB 11:30:56.962833 memdclient.go:188: Resolving response OP=0x89. Opaque=4
GOCB 11:30:56.962848 memdclient.go:121: Dispatching response callback. OP=0x89. Opaque=4
GOCB 11:30:56.962910 agent.go:527: Attempting to request CCCP configuration
GOCB 11:30:56.962954 memdclient.go:74: Writing request. [fd63:6f75:6368:1075:f964:44a2:2178:4cdd]:54836 OP=0xb5. Opaque=5
GOCB 11:30:56.974463 memdclient.go:188: Resolving response OP=0xb5. Opaque=5
GOCB 11:30:56.974480 memdclient.go:121: Dispatching response callback. OP=0xb5. Opaque=5
GOCB 11:30:56.976317 agent.go:562: Successfully connected
GOCB 11:30:56.976347 agentrouting.go:216: Switching routing data (update)...
GOCB 11:30:56.976420 agentrouting.go:217: New Routing Data:
Revision ID: 32
Client Multiplexer: Pipeline 0:
Address: fd63:6f75:6368:2068:c8e3:a1ff:fed9:38ca:11210
Max Clients: 1
Num Clients: 0
Max Items: 2048
Op Queue:
Num Items: 0
Is Open: true
Dead Pipeline:
Dead-Server Queue
Op Queue:
Num Items: 0
Is Open: true

Capi Eps:

• http://fd63:6f75:6368:2068:c8e3:a1ff:fed9:38ca:8092/travel-sample
  Mgmt Eps:
• http://fd63:6f75:6368:2068:c8e3:a1ff:fed9:38ca:8091
  N1ql Eps:
• http://fd63:6f75:6368:2068:c8e3:a1ff:fed9:38ca:8093
  FTS Eps:
• http://fd63:6f75:6368:2068:c8e3:a1ff:fed9:38ca:8094
  Source Data: *
  GOCB 11:30:56.976561 memdclient.go:158: memdClient read failure: read tcp [fd63:6f75:6368:1075:f964:44a2:2178:4cdd]:54836->[fd63:6f75:6368:2068:c8e3:a1ff:fed9:38ca]:11210: use of closed network connection
  GOCB 11:30:56.976606 memdclient.go:196: Failed to shut down client connection (close tcp [fd63:6f75:6368:1075:f964:44a2:2178:4cdd]:54836->[fd63:6f75:6368:2068:c8e3:a1ff:fed9:38ca]:11210: use of closed network connection)
  GOCB 11:30:56.976595 memdpipeline.go:266: Pipeline Client `0xc42022bea0` preparing for new client loop
  GOCB 11:30:56.976582 agent.go:429: CCCP Looper starting.
  GOCB 11:30:56.976641 memdpipeline.go:273: Pipeline Client `0xc42022bea0` is on parent: 0xc42007d220
  GOCB 11:30:56.976653 memdpipeline.go:279: Pipeline Client `0xc42022bea0` retrieving new client connection
  GOCB 11:30:56.976720 agentrouting.go:41: Failed to connect. dial tcp: address fd63:6f75:6368:2068:c8e3:a1ff:fed9:38ca:11210: too many colons in address
  GOCB 11:30:56.976733 memdpipeline.go:266: Pipeline Client `0xc42022bea0` preparing for new client loop
  GOCB 11:30:56.976739 memdpipeline.go:273: Pipeline Client `0xc42022bea0` is on parent: 0xc42007d220
  GOCB 11:30:56.976745 memdpipeline.go:279: Pipeline Client `0xc42022bea0` retrieving new client connection
  Error while Upsert:operation has timed out
  Error while Get:operation has timed out
  GOCB 11:31:01.977622 agentrouting.go:41: Failed to connect. dial tcp: address fd63:6f75:6368:2068:c8e3:a1ff:fed9:38ca:11210: too many colons in address
  GOCB 11:31:01.977673 memdpipeline.go:266: Pipeline Client `0xc42022bea0` preparing for new client loop
  GOCB 11:31:01.977688 memdpipeline.go:273: Pipeline Client `0xc42022bea0` is on parent: 0xc42007d220
  GOCB 11:31:01.977698 memdpipeline.go:279: Pipeline Client `0xc42022bea0` retrieving new client connection
  Error while Get:operation has timed out
  Error while Upsert:operation has timed out
  GOCB 11:31:02.477849 agent.go:465: CCCPPOLL: Failed to retrieve CCCP config. operation has timed out
  GOCB 11:31:02.477879 agent.go:486: CCCPPOLL: Failed to retrieve config from any node.
  Error while Get:operation has timed out
  Error while Upsert:operation has timed out
  GOCB 11:31:06.978211 agentrouting.go:41: Failed to connect. dial tcp: address fd63:6f75:6368:2068:c8e3:a1ff:fed9:38ca:11210: too many colons in address
  GOCB 11:31:06.978269 memdpipeline.go:266: Pipeline Client `0xc42022bea0` preparing for new client loop
  GOCB 11:31:06.978298 memdpipeline.go:273: Pipeline Client `0xc42022bea0` is on parent: 0xc42007d220
  GOCB 11:31:06.978319 memdpipeline.go:279: Pipeline Client `0xc42022bea0` retrieving new client connection
  Error while Upsert:operation has timed out
  Error while Get:operation has timed out
  GOCB 11:31:07.980166 agent.go:465: CCCPPOLL: Failed to retrieve CCCP config. operation has timed out
  GOCB 11:31:07.980207 agent.go:486: CCCPPOLL: Failed to retrieve config from any node.

Process finished with exit code 130 (interrupted by signal 2: SIGINT)