Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
Cheshire-Cat
-
Untriaged
-
-
1
-
Unknown
Description
Build : 7.0.0-2635
Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
Scale : 3
Iteration : 1st
All rebalance operations in the longevity system test are failing with the following errors -
[ns_server:error,2020-07-21T13:52:53.934-07:00,ns_1@172.23.97.74:<0.22710.290>:ns_single_vbucket_mover:spawn_and_wait:82]Got unexpected exit signal {'EXIT',<0.20904.290>,
|
{{bulk_set_vbucket_state_failed,
|
[{'ns_1@172.23.120.81',
|
{'EXIT',
|
{{{{{badmatch,
|
[{<26100.16216.43>,
|
{done,exit,
|
{socket_closed,
|
{gen_server,call,
|
[<26100.16335.43>,
|
{setup_streams,[1023]},
|
infinity]}},
|
[{gen_server,call,3,
|
[{file,"gen_server.erl"},
|
{line,214}]},
|
{dcp_replicator,
|
'-spawn_and_wait/1-fun-0-',1,
|
[{file,"src/dcp_replicator.erl"},
|
{line,243}]}]}}]},
|
[{misc,
|
sync_shutdown_many_i_am_trapping_exits,
|
1,
|
[{file,"src/misc.erl"},{line,1374}]},
|
{dcp_replicator,spawn_and_wait,1,
|
[{file,"src/dcp_replicator.erl"},
|
{line,265}]},
|
{dcp_replicator,handle_call,3,
|
[{file,"src/dcp_replicator.erl"},
|
{line,121}]},
|
{gen_server,try_handle_call,4,
|
[{file,"gen_server.erl"},{line,636}]},
|
{gen_server,handle_msg,6,
|
[{file,"gen_server.erl"},{line,665}]},
|
{proc_lib,init_p_do_apply,3,
|
[{file,"proc_lib.erl"},{line,247}]}]},
|
{gen_server,call,
|
[<26100.16101.43>,
|
{setup_replication,[1023]},
|
infinity]}},
|
{gen_server,call,
|
['replication_manager-WAREHOUSE',
|
{change_vbucket_replication,1023,
|
'ns_1@172.23.97.74'},
|
infinity]}},
|
{gen_server,call,
|
[{'janitor_agent-WAREHOUSE',
|
'ns_1@172.23.120.81'},
|
{if_rebalance,<0.22526.290>,
|
{update_vbucket_state,913,replica,
|
undefined,'ns_1@172.23.96.14'}},
|
infinity]}}}}]},
|
[{janitor_agent,bulk_set_vbucket_state,4,
|
[{file,"src/janitor_agent.erl"},{line,403}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,232}]}]}}
|
[ns_server:error,2020-07-21T13:52:53.934-07:00,ns_1@172.23.97.74:<0.21570.290>:ns_single_vbucket_mover:spawn_and_wait:82]Got unexpected exit signal {'EXIT',<0.22588.290>,
|
{{bulk_set_vbucket_state_failed,
|
[{'ns_1@172.23.120.81',
|
{'EXIT',
|
{{{{{badmatch,
|
[{<26100.16216.43>,
|
{done,exit,
|
{socket_closed,
|
{gen_server,call,
|
[<26100.16335.43>,
|
{setup_streams,[1023]},
|
infinity]}},
|
[{gen_server,call,3,
|
[{file,"gen_server.erl"},
|
{line,214}]},
|
{dcp_replicator,
|
'-spawn_and_wait/1-fun-0-',1,
|
[{file,"src/dcp_replicator.erl"},
|
{line,243}]}]}}]},
|
[{misc,
|
sync_shutdown_many_i_am_trapping_exits,
|
1,
|
[{file,"src/misc.erl"},{line,1374}]},
|
{dcp_replicator,spawn_and_wait,1,
|
[{file,"src/dcp_replicator.erl"},
|
{line,265}]},
|
{dcp_replicator,handle_call,3,
|
[{file,"src/dcp_replicator.erl"},
|
{line,121}]},
|
{gen_server,try_handle_call,4,
|
[{file,"gen_server.erl"},{line,636}]},
|
{gen_server,handle_msg,6,
|
[{file,"gen_server.erl"},{line,665}]},
|
{proc_lib,init_p_do_apply,3,
|
[{file,"proc_lib.erl"},{line,247}]}]},
|
{gen_server,call,
|
[<26100.16101.43>,
|
{setup_replication,[1023]},
|
...
|
...
|
...
|
On 172.23.120.81, at the same time the following msgs can be seen in the memcached logs -
2020-07-21T13:52:52.200735-07:00 INFO 67: Client 172.23.104.254:56086 authenticated as <ud>Administrator</ud>
|
2020-07-21T13:52:52.237301-07:00 INFO 67: HELO [{"a":"couchbase-java-client/${project.version} (git: ${git.commit.id.describe}, core: ${git.commit.id.describe}) (Linux/3.10.0-1062.9.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_252-b09)","i":"323CC12376D83AC6/00000000116FA20A"}] TCP nodelay, Mutation seqno, XATTR, XERROR, Select bucket [ 172.23.104.254:46662 - 172.23.120.81:11210 (not authenticated) ]
|
2020-07-21T13:52:52.256367-07:00 INFO 67: Client 172.23.104.254:46662 authenticated as <ud>Administrator</ud>
|
2020-07-21T13:52:52.382994-07:00 INFO 67: Client 127.0.0.1:47011 authenticated as <ud>@ns_server</ud>
|
2020-07-21T13:52:52.383340-07:00 INFO 67: HELO [regular] [ 127.0.0.1:47011 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]
|
2020-07-21T13:52:52.639734-07:00 INFO (WAREHOUSE) EventuallyPersistentEngine::enableTraffic() result true
|
2020-07-21T13:52:52.639752-07:00 INFO (WAREHOUSE) EventuallyPersistentEngine::handleTrafficControlCmd() Data traffic to persistence engine is enabled
|
2020-07-21T13:52:52.954954-07:00 INFO (WAREHOUSE) VBucket: created vb:1023 with state:pending initialState:dead lastSeqno:0 persistedRange:{0,0} max_cas:0 uuid:162031797748993 topology:null
|
2020-07-21T13:52:52.984729-07:00 INFO 68: Client 127.0.0.1:56863 authenticated as <ud>@ns_server</ud>
|
2020-07-21T13:52:52.985601-07:00 INFO 68: HELO [proxy] XATTR, Snappy, JSON, Collections [ 127.0.0.1:56863 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]
|
2020-07-21T13:52:53.023665-07:00 INFO (WAREHOUSE) EventuallyPersistentEngine::dcpOpen: opening new DCP Consumer handler - stream name:replication:ns_1@172.23.97.74->ns_1@172.23.120.81:WAREHOUSE, opaque:0, seqno:0, flags:0b00000000000000000000000100100100 value:{"consumer_name":"ns_1@172.23.120.81"}
|
2020-07-21T13:52:53.023690-07:00 INFO 68: DCP connection opened successfully. INCLUDE_XATTRS, DELETE_TIMES, INCLUDE_DELETED_USER_XATTRS [ 127.0.0.1:56863 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]
|
2020-07-21T13:52:53.043747-07:00 INFO 68: (WAREHOUSE) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.97.74->ns_1@172.23.120.81:WAREHOUSE - (vb:1023) Attempting to add stream: opaque_:1, start_seqno_:0, end_seqno_:18446744073709551615, vb_uuid:162031797748993, snap_start_seqno_:0, snap_end_seqno_:0, last_seqno:0, stream_req_value:{"uid":"0"}
|
2020-07-21T13:52:53.056771-07:00 WARNING 68: exception occurred in runloop during packet execution. Closing connection: xattr::utils::check_len(1592262656) exceeds 80. Cookies: [{"aiostat":"success","connection":"[ 127.0.0.1:56863 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f98280ef510","ewouldblock":false,"packet":{"bodylen":120,"cas":1595354098397937664,"datatype":["Snappy","Xattr"],"extlen":21,"key":"<ud>.8DE6199D-102_22050</ud>","keylen":19,"magic":"ClientRequest","opaque":1,"opcode":"DCP_DELETION","vbucket":1023},"refcount":1}]
|
2020-07-21T13:52:53.056847-07:00 INFO 68: (No Engine) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.97.74->ns_1@172.23.120.81:WAREHOUSE - Removing connection [ 127.0.0.1:56863 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]
|
2020-07-21T13:52:53.056980-07:00 WARNING 68: (WAREHOUSE) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.97.74->ns_1@172.23.120.81:WAREHOUSE - (vb:1023) Setting stream to dead state, last_seqno is 371, unAckedBytes is 0, status is The stream closed early because the conn was disconnected
|
2020-07-21T13:52:53.235599-07:00 INFO 69: HELO [{"a":"couchbase-java-client/${project.version} (git: ${git.commit.id.describe}, core: ${git.commit.id.describe}) (Linux/3.10.0-1062.9.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_252-b09)","i":"012959CB310744C8/000000004031D9B5"}] TCP nodelay, Mutation seqno, XATTR, XERROR, Select bucket [ 172.23.104.254:56134 - 172.23.120.81:11210 (not authenticated) ]
|
2020-07-21T13:52:53.272656-07:00 INFO 69: Client 172.23.104.254:56134 authenticated as <ud>Administrator</ud>
|
2020-07-21T13:52:53.280790-07:00 INFO 66: [ 127.0.0.1:44114 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ] Updated cluster configuration for bucket [WAREHOUSE]. New revision: 2262
|
2020-07-21T13:52:53.280872-07:00 INFO Pushing new cluster config for bucket:[WAREHOUSE] revision:[2262]
|
2020-07-21T13:52:53.285271-07:00 INFO 69: HELO [{"a":"couchbase-java-client/${project.version} (git: ${git.commit.id.describe}, core: ${git.commit.id.describe}) (Linux/3.10.0-1062.9.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_252-b09)","i":"323CC12376D83AC6/FFFFFFFFDC9D09FF"}] TCP nodelay, Mutation seqno, XATTR, XERROR, Select bucket [ 172.23.104.254:46704 - 172.23.120.81:11210 (not authenticated) ]
|
2020-07-21T13:52:53.306776-07:00 INFO 69: Client 172.23.104.254:46704 authenticated as <ud>Administrator</ud>
|
Attachments
Issue Links
- duplicates
-
-
- Closed
-