Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
6.5.0
-
Centos1
-
Triaged
-
-
Yes
-
KV-Engine Mad-Hatter GA
Description
Build: 6.5.0-4908 not seen on 4890
Test: MH longevity with durability
Cycle: 2nd
Day: 1st
Seeing following crashÂ
Service 'memcached' exited with status 134. Restarting. Messages:
|
2019-11-29T21:06:49.818637-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f83e4b3d000+0x8f213]
|
2019-11-29T21:06:49.818649-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f83df688000+0x74098]
|
2019-11-29T21:06:49.818656-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f83df688000+0x77434]
|
2019-11-29T21:06:49.818665-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f83df688000+0x77843]
|
2019-11-29T21:06:49.818671-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f83df688000+0x77924]
|
2019-11-29T21:06:49.818677-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f83df688000+0x809f9]
|
2019-11-29T21:06:49.818687-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f83df688000+0x12f964]
|
2019-11-29T21:06:49.818691-08:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f83e69e6000+0x8ee7]
|
2019-11-29T21:06:49.818695-08:00 CRITICAL /lib64/libpthread.so.0() [0x7f83e4408000+0x7dc5]
|
2019-11-29T21:06:49.818720-08:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f83e4047000+0xf621d]Â hidens_log 000ns_1@172.23.99.25 9:06:52 PMÂ Â Â 29 Nov, 2019
|
|
Test stepÂ
[2019-11-29T20:28:41-08:00, sequoiatools/couchbase-cli:6.5:0c44c4] server-add -c 172.23.108.103:8091 --server-add https://172.23.106.100 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
|
[2019-11-29T20:42:41-08:00, sequoiatools/couchbase-cli:6.5:4b9505] failover -c 172.23.108.103:8091 --server-failover 172.23.104.5:8091 -u Administrator -p password
|
[2019-11-29T21:03:27-08:00, sequoiatools/couchbase-cli:6.5:b143b9] failover -c 172.23.108.103:8091 --server-failover 172.23.99.20:8091 -u Administrator -p password --force
|
[2019-11-29T21:04:21-08:00, sequoiatools/couchbase-cli:6.5:ba2ca6] rebalance -c 172.23.108.103:8091 -u Administrator -p password
|
→Â
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
|
|
|
docker logs ba2ca6
|
docker start ba2ca6
|
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[2019-11-29T21:07:15-08:00, sequoiatools/cmd:1e7d4b] 60
|
Rebalance failed with during same periodÂ
[user:error,2019-11-29T21:06:52.374-08:00,ns_1@172.23.108.103:<0.12064.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {mover_crashed,
|
               {unexpected_exit,
|
                {'EXIT',<0.7202.822>,
|
                {{{{{child_interrupted,
|
                   {'EXIT',<25874.23227.201>,socket_closed}},
|
                  [{dcp_replicator,spawn_and_wait,1,
|
                   [{file,"src/dcp_replicator.erl"},
|
                    {line,266}]},
|
                   {dcp_replicator,handle_call,3,
|
                   [{file,"src/dcp_replicator.erl"},
|
                    {line,121}]},
|
                   {gen_server,try_handle_call,4,
|
                   [{file,"gen_server.erl"},{line,636}]},
|
                   {gen_server,handle_msg,6,
|
                   [{file,"gen_server.erl"},{line,665}]},
|
                   {proc_lib,init_p_do_apply,3,
|
                   [{file,"proc_lib.erl"},{line,247}]}]},
|
                  {gen_server,call,
|
                  [<25874.23258.201>,
|
                   {setup_replication,
|
                   [21,22,106,163,336,760,939,940,941,942,
|
                    943,944,945]},
|
                   infinity]}},
|
                 {gen_server,call,
|
                  ['replication_manager-STOCK',
|
                  {change_vbucket_replication,21,
|
                   'ns_1@172.23.99.25'},
|
                  infinity]}},
|
                 {gen_server,call,
|
                 [{'janitor_agent-STOCK',
|
                  'ns_1@172.23.104.61'},
|
                  {if_rebalance,<0.5737.821>,
|
                  {update_vbucket_state,47,active,
|
                   undefined,undefined,
|
                   [['ns_1@172.23.104.61',
|
                    'ns_1@172.23.97.122']]}},
|
                  infinity]}}}}}.
|
Rebalance Operation Id = e3a4b107de62a1338edd76e84547484d
|
Attachments
Issue Links
Activity
Field | Original Value | New Value |
---|---|---|
Assignee | Trond Norbye [ trond ] | Daniel Owen [ owend ] |
Summary | [System test]: memcached exited with 134 | [System test]: Disk Checkpoint does not have an initialised HCS |
Assignee | Daniel Owen [ owend ] | Ben Huddleston [ ben.huddleston ] |
Due Date | 03/Dec/19 |
Sprint | KV-Engine Mad-Hatter GA [ 910 ] |
Rank | Ranked higher |
Triage | Untriaged [ 10351 ] | Triaged [ 10350 ] |
Assignee | Ben Huddleston [ ben.huddleston ] | Dave Rigby [ drigby ] |
Status | Open [ 1 ] | In Progress [ 3 ] |
Component/s | couchbase-bucket [ 10173 ] | |
Component/s | memcached [ 11621 ] |
Labels | system-test | approved-for-mad-hatter system-test |
Link | This issue blocks MB-36676 [ MB-36676 ] |
Assignee | Dave Rigby [ drigby ] | Vikas Chaudhary [ vikas.chaudhary ] |
Resolution | Fixed [ 1 ] | |
Status | In Progress [ 3 ] | Resolved [ 5 ] |
VERIFICATION STEPS |
closed based on - |
|
Status | Resolved [ 5 ] | Closed [ 6 ] |
Remote Link | This issue links to "Page (Couchbase, Inc. Wiki)" [ 18931 ] |