Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37103

[System test]: Disk Checkpoint does not have an initialised HCS

    XMLWordPrintable

Details

    Description

      Build: 6.5.0-4908 not seen on 4890

      Test: MH longevity with durability

      Cycle: 2nd

      Day: 1st

      Seeing following crash 

      Service 'memcached' exited with status 134. Restarting. Messages:
      2019-11-29T21:06:49.818637-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f83e4b3d000+0x8f213]
      2019-11-29T21:06:49.818649-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f83df688000+0x74098]
      2019-11-29T21:06:49.818656-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f83df688000+0x77434]
      2019-11-29T21:06:49.818665-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f83df688000+0x77843]
      2019-11-29T21:06:49.818671-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f83df688000+0x77924]
      2019-11-29T21:06:49.818677-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f83df688000+0x809f9]
      2019-11-29T21:06:49.818687-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f83df688000+0x12f964]
      2019-11-29T21:06:49.818691-08:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f83e69e6000+0x8ee7]
      2019-11-29T21:06:49.818695-08:00 CRITICAL /lib64/libpthread.so.0() [0x7f83e4408000+0x7dc5]
      2019-11-29T21:06:49.818720-08:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f83e4047000+0xf621d] hidens_log 000ns_1@172.23.99.25 9:06:52 PM   29 Nov, 2019
      
      

      Test step 

      [2019-11-29T20:28:41-08:00, sequoiatools/couchbase-cli:6.5:0c44c4] server-add -c 172.23.108.103:8091 --server-add https://172.23.106.100 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2019-11-29T20:42:41-08:00, sequoiatools/couchbase-cli:6.5:4b9505] failover -c 172.23.108.103:8091 --server-failover 172.23.104.5:8091 -u Administrator -p password
      [2019-11-29T21:03:27-08:00, sequoiatools/couchbase-cli:6.5:b143b9] failover -c 172.23.108.103:8091 --server-failover 172.23.99.20:8091 -u Administrator -p password --force
      [2019-11-29T21:04:21-08:00, sequoiatools/couchbase-cli:6.5:ba2ca6] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      → 
       
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
       
       
      docker logs ba2ca6
      docker start ba2ca6
       
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2019-11-29T21:07:15-08:00, sequoiatools/cmd:1e7d4b] 60 

      Rebalance failed with during same period 

       [user:error,2019-11-29T21:06:52.374-08:00,ns_1@172.23.108.103:<0.12064.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {mover_crashed,
                                    {unexpected_exit,
                                     {'EXIT',<0.7202.822>,
                                      {{{{{child_interrupted,
                                           {'EXIT',<25874.23227.201>,socket_closed}},
                                          [{dcp_replicator,spawn_and_wait,1,
                                            [{file,"src/dcp_replicator.erl"},
                                             {line,266}]},
                                           {dcp_replicator,handle_call,3,
                                            [{file,"src/dcp_replicator.erl"},
                                             {line,121}]},
                                           {gen_server,try_handle_call,4,
                                            [{file,"gen_server.erl"},{line,636}]},
                                           {gen_server,handle_msg,6,
                                            [{file,"gen_server.erl"},{line,665}]},
                                           {proc_lib,init_p_do_apply,3,
                                            [{file,"proc_lib.erl"},{line,247}]}]},
                                         {gen_server,call,
                                          [<25874.23258.201>,
                                           {setup_replication,
                                            [21,22,106,163,336,760,939,940,941,942,
                                             943,944,945]},
                                           infinity]}},
                                        {gen_server,call,
                                         ['replication_manager-STOCK',
                                          {change_vbucket_replication,21,
                                           'ns_1@172.23.99.25'},
                                          infinity]}},
                                       {gen_server,call,
                                        [{'janitor_agent-STOCK',
                                          'ns_1@172.23.104.61'},
                                         {if_rebalance,<0.5737.821>,
                                          {update_vbucket_state,47,active,
                                           undefined,undefined,
                                           [['ns_1@172.23.104.61',
                                             'ns_1@172.23.97.122']]}},
                                         infinity]}}}}}.
      Rebalance Operation Id = e3a4b107de62a1338edd76e84547484d

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            vikas.chaudhary Vikas Chaudhary created issue -
            owend Daniel Owen made changes -
            Field Original Value New Value
            Assignee Trond Norbye [ trond ] Daniel Owen [ owend ]
            owend Daniel Owen made changes -
            Summary [System test]: memcached exited with 134 [System test]: Disk Checkpoint does not have an initialised HCS
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Ben Huddleston [ ben.huddleston ]
            owend Daniel Owen made changes -
            Due Date 03/Dec/19
            owend Daniel Owen made changes -
            Sprint KV-Engine Mad-Hatter GA [ 910 ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            owend Daniel Owen made changes -
            Triage Untriaged [ 10351 ] Triaged [ 10350 ]
            drigby Dave Rigby made changes -
            Link This issue is duplicated by MB-37109 [ MB-37109 ]
            drigby Dave Rigby made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Dave Rigby [ drigby ]
            drigby Dave Rigby made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            drigby Dave Rigby made changes -
            Component/s couchbase-bucket [ 10173 ]
            Component/s memcached [ 11621 ]
            lynn.straus Lynn Straus made changes -
            Labels system-test approved-for-mad-hatter system-test
            lynn.straus Lynn Straus made changes -
            Link This issue blocks MB-36676 [ MB-36676 ]
            drigby Dave Rigby made changes -
            Assignee Dave Rigby [ drigby ] Vikas Chaudhary [ vikas.chaudhary ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            arunkumar Arunkumar Senthilnathan made changes -
            Link This issue duplicates MB-37161 [ MB-37161 ]
            ritam.sharma Ritam Sharma made changes -
            VERIFICATION STEPS closed based on - MB-37109
            Status Resolved [ 5 ] Closed [ 6 ]
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Remote Link This issue links to "Page (Couchbase, Inc. Wiki)" [ 18931 ]

            People

              vikas.chaudhary Vikas Chaudhary
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty