Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Incomplete
Priority: Blocker
Fix Version/s: 2.10.6
Affects Version/s: 2.10.5
Component/s: library
Labels:
None

Story Points:
1

Description

Data mismatch starts happening for timers after swap rebalance of multiple kv nodes

 ./testrunner -i /tmp/testexec.16358.ini -p get-cbcollect-info=True,GROUP=n1ql_op_with_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,,handler_code=n1ql_op_with_timers,replicas=1,GROUP=n1ql_op_with_timers

Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 0 Expected : 10080  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'172.23.104.253': {}}

./testrunner -i /tmp/testexec.19911.ini -p get-cbcollect-info=True,GROUP=bucket_op_with_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,,handler_code=bucket_op_with_timers,GROUP=bucket_op_with_timers

Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 0 Expected : 10080  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'172.23.105.235': {}}

In the test, documents loading is triggered asynchronously before swap rebalance of the kv node.

2019-11-13 03:24:55 | INFO | MainProcess | test_thread | [rest_client.add_node] adding remote node @172.23.106.69:8091 to this cluster @172.23.106.74:8091

2019-11-13 03:24:56 | INFO | MainProcess | test_thread | [rest_client.add_node] adding remote node @172.23.106.67:8091 to this cluster @172.23.106.74:8091

2019-11-13 03:24:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.106.67 not part of cluster inactiveAdded

2019-11-13 03:24:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.106.69 not part of cluster inactiveAdded

2019-11-13 03:24:59 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.75:11210 src_bucket

2019-11-13 03:24:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.106.67 not part of cluster inactiveAdded

2019-11-13 03:24:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.106.69 not part of cluster inactiveAdded

2019-11-13 03:24:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.106.67 not part of cluster inactiveAdded

2019-11-13 03:24:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.106.69 not part of cluster inactiveAdded

2019-11-13 03:24:59 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.76:11210 src_bucket

2019-11-13 03:24:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.106.67 not part of cluster inactiveAdded

2019-11-13 03:24:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.106.69 not part of cluster inactiveAdded

2019-11-13 03:24:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.106.67 not part of cluster inactiveAdded

2019-11-13 03:24:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.106.69 not part of cluster inactiveAdded

2019-11-13 03:24:59 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.77:11210 src_bucket

2019-11-13 03:24:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.106.67 not part of cluster inactiveAdded

2019-11-13 03:24:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.106.69 not part of cluster inactiveAdded

2019-11-13 03:24:59 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.106.76,ns_1@172.23.106.77', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.106.69,ns_1@172.23.106.74,ns_1@172.23.106.67,ns_1@172.23.106.76,ns_1@172.23.106.73,ns_1@172.23.106.75,ns_1@172.23.106.77'}

2019-11-13 03:24:59 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started

2019-11-13 03:24:59 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 %

2019-11-13 03:24:59 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00%

2019-11-13 03:25:02 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 3.01206994057 seconds

2019-11-13 03:25:09 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 11.12 %

2019-11-13 03:25:09 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 11.12%

2019-11-13 03:25:20 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 19.05 %

2019-11-13 03:25:20 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 19.05%

2019-11-13 03:25:30 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 26.66 %

2019-11-13 03:25:30 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 26.66%

2019-11-13 03:25:40 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 40.04 %

2019-11-13 03:25:40 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 40.04%

2019-11-13 03:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 42.86 %

2019-11-13 03:25:50 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 42.86%

2019-11-13 03:26:00 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 55.30 %

2019-11-13 03:26:00 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 55.30%

2019-11-13 03:26:10 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 66.67 %

2019-11-13 03:26:10 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 66.67%

2019-11-13 03:26:20 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 71.00 %

2019-11-13 03:26:20 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 71.00%

2019-11-13 03:26:30 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 78.57 %

2019-11-13 03:26:30 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 78.57%

2019-11-13 03:26:40 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00%

2019-11-13 03:26:40 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.76:8091/pools/default body:  headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password

In the eventing handler code, timer is suppose to trigger after 30 seconds but lcb authentication is failed due to which eventing is unable to set timer.

During that time, Eventing log is filled with LCB_AUTH_ERROR

86554:2019-11-13T03:25:01.855-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86558:2019-11-13T03:25:01.906-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86573:2019-11-13T03:25:01.915-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86595:2019-11-13T03:25:01.963-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_955883421.sock:94849] [lcb,server L:777 I:2248989486] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f25c4077220,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86599:2019-11-13T03:25:01.982-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86612:2019-11-13T03:25:02.005-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86617:2019-11-13T03:25:02.010-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_955883421.sock:94855] [lcb,server L:777 I:823726560] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7fdd8c0ff3f0,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86619:2019-11-13T03:25:02.015-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_955883421.sock:94849] [lcb,server L:777 I:2248989486] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f25c4077220,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86622:2019-11-13T03:25:02.021-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_955883421.sock:94849] [lcb,server L:777 I:2248989486] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f25c4077220,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86643:2019-11-13T03:25:02.058-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_955883421.sock:94855] [lcb,server L:777 I:823726560] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7fdd8c0ff3f0,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86646:2019-11-13T03:25:02.060-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_955883421.sock:94855] [lcb,server L:777 I:823726560] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7fdd8c0ff3f0,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86647:2019-11-13T03:25:02.061-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86650:2019-11-13T03:25:02.063-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86678:2019-11-13T03:25:02.090-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_955883421.sock:94849] [lcb,server L:777 I:2248989486] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f25c4077220,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86691:2019-11-13T03:25:02.128-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_955883421.sock:94855] [lcb,server L:777 I:823726560] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7fdd8c0ff3f0,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86695:2019-11-13T03:25:02.156-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86698:2019-11-13T03:25:02.161-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86718:2019-11-13T03:25:02.181-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_955883421.sock:94849] [lcb,server L:777 I:2248989486] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f25c4077220,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86721:2019-11-13T03:25:02.195-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_955883421.sock:94849] [lcb,server L:777 I:2248989486] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f25c4077220,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86732:2019-11-13T03:25:02.227-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_955883421.sock:94855] [lcb,server L:777 I:823726560] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7fdd8c0ff3f0,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86735:2019-11-13T03:25:02.238-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_955883421.sock:94855] [lcb,server L:777 I:823726560] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7fdd8c0ff3f0,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86746:2019-11-13T03:25:02.272-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86749:2019-11-13T03:25:02.276-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86760:2019-11-13T03:25:02.303-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_955883421.sock:94849] [lcb,server L:777 I:2248989486] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f25c4077220,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86763:2019-11-13T03:25:02.305-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_955883421.sock:94849] [lcb,server L:777 I:2248989486] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f25c4077220,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86774:2019-11-13T03:25:02.339-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_955883421.sock:94855] [lcb,server L:777 I:823726560] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7fdd8c0ff3f0,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86777:2019-11-13T03:25:02.341-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_955883421.sock:94855] [lcb,server L:777 I:823726560] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7fdd8c0ff3f0,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86788:2019-11-13T03:25:02.396-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86791:2019-11-13T03:25:02.399-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86810:2019-11-13T03:25:02.426-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_955883421.sock:94849] [lcb,server L:777 I:2248989486] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f25c4077220,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86813:2019-11-13T03:25:02.428-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_955883421.sock:94849] [lcb,server L:777 I:2248989486] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f25c4077220,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86824:2019-11-13T03:25:02.471-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_955883421.sock:94855] [lcb,server L:777 I:823726560] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7fdd8c0ff3f0,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86827:2019-11-13T03:25:02.479-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_955883421.sock:94855] [lcb,server L:777 I:823726560] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7fdd8c0ff3f0,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86838:2019-11-13T03:25:02.538-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86841:2019-11-13T03:25:02.541-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86860:2019-11-13T03:25:02.574-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_955883421.sock:94849] [lcb,server L:777 I:2248989486] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f25c4077220,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86863:2019-11-13T03:25:02.580-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_955883421.sock:94849] [lcb,server L:777 I:2248989486] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f25c4077220,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86874:2019-11-13T03:25:02.619-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_955883421.sock:94855] [lcb,server L:777 I:823726560] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7fdd8c0ff3f0,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86882:2019-11-13T03:25:02.624-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_955883421.sock:94855] [lcb,server L:777 I:823726560] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7fdd8c0ff3f0,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

86903:2019-11-13T03:25:02.694-08:00 [Info] eventing-consumer [worker_Function_28066170_test_eventing_rebalance_with_multiple_kv_nodes_2:/tmp/127.0.0.1:8091_2_955883421.sock:94868] [lcb,server L:777 I:2767056025] <NOHOST:NOPORT> (CTX=(nil),,SRV=0x7f9c7c056620,IX=0) Connection attempt failed. Received LCB_AUTH_ERROR (0x02) from libcouchbase, received 0 from operating system

memcahced reported invalid password

memcached.log.000002.txt:2019-11-13T03:25:01.855076-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[0951d172-e444-4d52-1695-0da9d3a3b70a] memcached.log.000002.txt:2019-11-13T03:25:01.905298-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[a474b816-807f-4241-65b9-50204a4edbaf] memcached.log.000002.txt:2019-11-13T03:25:01.914388-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[8c84f2f7-427d-403f-8f77-6bab1ea8e515] memcached.log.000002.txt:2019-11-13T03:25:01.961682-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[e37cca2b-81b3-4c37-4ed9-73a0556badfd] memcached.log.000002.txt:2019-11-13T03:25:01.980954-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[56e6d40b-e8f9-43f2-b384-d0508c71ced8] memcached.log.000002.txt:2019-11-13T03:25:02.003642-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[055591f2-e4d2-4eab-929b-28d6c3233ed4] memcached.log.000002.txt:2019-11-13T03:25:02.010237-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[b1f70e06-1479-406a-39dc-96eafe195b33] memcached.log.000002.txt:2019-11-13T03:25:02.010752-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[f4caeff2-fe4d-4996-813a-f04aa283d857] memcached.log.000002.txt:2019-11-13T03:25:02.020544-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[6b3c3f47-d70e-45dd-7e9d-2a4eac9da8bc] memcached.log.000002.txt:2019-11-13T03:25:02.058054-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[03e50aa6-298c-4f19-8927-8a25217e39d7] memcached.log.000002.txt:2019-11-13T03:25:02.060075-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[f8722235-3730-43fc-f75b-62b6e088db87] memcached.log.000002.txt:2019-11-13T03:25:02.060274-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[7ba18171-f35d-44ea-c832-66e379cbc803] memcached.log.000002.txt:2019-11-13T03:25:02.062806-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[f9a03a45-1e2d-494d-71de-145248084b5c] memcached.log.000002.txt:2019-11-13T03:25:02.087241-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[6ce774f6-3334-4ff3-24f0-496b622db8db] memcached.log.000002.txt:2019-11-13T03:25:02.127632-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[cd99e289-df43-42a6-4f02-20497ad14340] memcached.log.000002.txt:2019-11-13T03:25:02.156452-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[0538c38f-c5fd-4050-bf41-a5379e847dbd] memcached.log.000002.txt:2019-11-13T03:25:02.160161-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[671f814f-0a0c-4d2a-a643-d6cd57cfc2bc] memcached.log.000002.txt:2019-11-13T03:25:02.180398-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[c37d6f8f-2fdb-41c2-04db-1e5f9d4666e8] memcached.log.000002.txt:2019-11-13T03:25:02.193727-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[dfbd800e-216c-49ff-976c-096239da68fe] memcached.log.000002.txt:2019-11-13T03:25:02.225690-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[c143d783-3dee-4279-73c0-47f721bede52] memcached.log.000002.txt:2019-11-13T03:25:02.233399-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[7327c9b2-814f-4749-ee2f-0315475c2877] memcached.log.000002.txt:2019-11-13T03:25:02.270654-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[7baa583b-29c1-4b0f-0219-3131d020d0bb] memcached.log.000002.txt:2019-11-13T03:25:02.276455-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[4e1987e3-2a21-4404-0871-36ca3ec63a36] memcached.log.000002.txt:2019-11-13T03:25:02.302918-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[01556332-ecfb-4950-60a6-33929aef899e] memcached.log.000002.txt:2019-11-13T03:25:02.304770-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[e99f4ec2-bafd-42c2-8ad3-cf1350e5eec9] memcached.log.000002.txt:2019-11-13T03:25:02.337058-08:00 WARNING 61: Invalid password specified for [<ud>@eventing</ud>] UUID:[473aecd7-0205-4c52-63f5-eb665e40f525] memcached.log.000002.txt:2019-11-13T03:25:02.341124-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[77dea468-9ccd-409c-7866-21801c727ca5] memcached.log.000002.txt:2019-11-13T03:25:02.396470-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[afc56437-86d1-491b-a4e6-bc0e6eba57c9] memcached.log.000002.txt:2019-11-13T03:25:02.398927-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[cb47e73c-4bf8-4610-1924-33cf7a62a0d7] memcached.log.000002.txt:2019-11-13T03:25:02.426198-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[766a8856-b2cd-4ef4-4046-e83165fe880a] memcached.log.000002.txt:2019-11-13T03:25:02.428360-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[ef06bf0d-debb-4156-cd8d-9a5490fb5e95] memcached.log.000002.txt:2019-11-13T03:25:02.471185-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[e15cc6d0-dfaf-4a9b-ede8-d64cd24e5d18] memcached.log.000002.txt:2019-11-13T03:25:02.478532-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[67675cec-eaac-46cb-8a6f-79930a740309] memcached.log.000002.txt:2019-11-13T03:25:02.538075-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[10ccf837-3fa2-404a-c8eb-7754a41fe7d4] memcached.log.000002.txt:2019-11-13T03:25:02.540548-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[a5e4a5f8-0e8a-4014-a4e8-5dd689be7f6c] memcached.log.000002.txt:2019-11-13T03:25:02.574465-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[ada1307f-7d50-4896-a7f6-99f1bff4969d] memcached.log.000002.txt:2019-11-13T03:25:02.580377-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[2c0df47a-92a2-48c8-8b29-eefe4b43f3f5] memcached.log.000002.txt:2019-11-13T03:25:02.611907-08:00 WARNING 61: Invalid password specified for [<ud>@eventing</ud>] UUID:[18ae8315-307b-4393-1079-5dba93604d2a] memcached.log.000002.txt:2019-11-13T03:25:02.623563-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[56752aab-3310-41c2-75f0-99dd3282c68f] memcached.log.000002.txt:2019-11-13T03:25:02.692173-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[0901389c-2ede-470a-b8cf-c031cac57427] memcached.log.000002.txt:2019-11-13T03:25:02.727378-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[2ac96a60-b6b4-471b-5888-84d225282df5] memcached.log.000002.txt:2019-11-13T03:25:02.729486-08:00 WARNING 54: Invalid password specified for [<ud>@eventing</ud>] UUID:[a21f4451-a4c5-47db-3313-002325445203] memcached.log.000002.txt:2019-11-13T03:25:02.779051-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[1d66f1d2-41bf-402a-b10b-e06733929f3a] memcached.log.000002.txt:2019-11-13T03:25:02.784393-08:00 WARNING 60: Invalid password specified for [<ud>@eventing</ud>] UUID:[977c98a5-5698-496c-e7ff-e6654e24a36b]

This issue starts happening from build 4712. There is no changes in eventing code from 4711 and 4712 build.
There is one change in libcouchbase which is related to caching credentials: http://review.couchbase.org/#/c/116958/

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

172.23.104.103-20191204-1148-diag.zip
40.85 MB
04/Dec/19 9:50 PM
172.23.104.105-20191204-1150-diag.zip
12.93 MB
04/Dec/19 9:49 PM
172.23.104.107-20191204-1151-diag.zip
12.23 MB
04/Dec/19 9:48 PM
172.23.104.109-20191204-1153-diag.zip
11.26 MB
04/Dec/19 9:48 PM
172.23.104.199-20191204-1155-diag.zip
11.37 MB
04/Dec/19 9:48 PM
172.23.104.201-20191204-1156-diag.zip
13.29 MB
04/Dec/19 9:49 PM
172.23.104.202-20191204-1158-diag.zip
5.24 MB
04/Dec/19 9:49 PM
capturedPackets.pcap
14 kB
13/Nov/19 7:00 AM
OriginalCaptured.pcap
66.49 MB
13/Nov/19 7:03 AM
test.log
361 kB
04/Dec/19 9:49 PM

Issue Links

blocks

MB-36326 [System test]: Eventing rebalance hung since 9 hrs

Closed

MB-36767 [System test]: service_rebalance_failed,eventing, worker died, timed out

Closed

causes

MB-36793 Data mismatch for rebalance in/out multiple kv nodes

Closed

relates to

CCBC-1260 Need way to boostrap using cluster mgmt port

Resolved

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

For Gerrit Dashboard: CCBC-1120
#	Subject	Branch	Project	Status	CR	V
118289,2	CCBC-1120: Use credentials cache only for N1QL queries	release-2.10	libcouchbase	Status: MERGED	+2	+1
122903,4	MB-37564: Update libcouchbase to 2.10.6	master	manifest	Status: MERGED	+2	+1

Activity

People

Assignee:: Ankit Prabhu

Reporter:: Ankit Prabhu

Votes:: 0 Vote for this issue

Watchers:: 11 Start watching this issue

Dates

Created:: 13/Nov/19 6:00 AM

Updated:: 08/Apr/21 7:27 AM

Resolved:: 08/Apr/21 7:27 AM

Gerrit Reviews

There are no open Gerrit changes

Show There are 2 closed Gerrit changes

Hide There are 2 closed Gerrit changes

CCBC-1120: Use credentials cache only for N1QL queries: Gerrit Review:

MB-37564: Update libcouchbase to 2.10.6: Gerrit Review:

Continuous LCB_AUTH_ERROR (0x02) after kv swap rebalance when bootstrapped via ns_server mgmt port

Details

Description

Attachments

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty