Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-16223

[system test]Most data nodes are unresponsive on a cluster reboot

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Incomplete
    • Major
    • 4.0.0
    • 4.0.0
    • ns_server
    • Security Level: Public
    • 400-4049

    Description

      1. Setup Cluster + Indexes
      2. Load data
      3. Reboot all nodes - Noticed all but the data nodes have come back.

      Attached screenshot from current cluster.

      Waited for around 2 hours for the nodes to be up, but seeing no progress.

      The beam, memcached processes are up, but the node is in pending state.

      top
      Mem:  23928632k total, 20588840k used,  3339792k free,    93624k buffers
      Swap: 10829816k total,    20464k used, 10809352k free,  3726788k cached
       
        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                     
       1195 couchbas  20   0 3022m 1.0g 3764 S 46.7  4.5  58:49.67 beam.smp                                                                                     
       1408 couchbas  20   0 15.0g  14g 6996 S 13.6 63.5  19:45.61 memcached                                                                                    
       1263 couchbas  20   0 1048m  85m 6180 S  1.3  0.4   1:43.88 beam.smp                                                                                     
       1371 couchbas  20   0  365m 7468 4712 S  0.7  0.0   0:40.46 goxdcr                                                                                       
          4 root      20   0     0    0    0 S  0.3  0.0   0:00.78 ksoftirqd/0                                                                                  
       1112 couchbas  20   0 1276m  16m 2748 S  0.3  0.1   0:31.23 beam.smp                                                                                     
       1356 couchbas  20   0  4944 1684 1132 S  0.3  0.0   0:21.75 goport            
      

      Logs

      https://s3.amazonaws.com/cb-customers/1/1/collectinfo-2015-09-03T230823-ns_1%4010.6.2.164.zip
      https://s3.amazonaws.com/cb-customers/1/1/collectinfo-2015-09-03T230823-ns_1%4010.6.2.194.zip
      https://s3.amazonaws.com/cb-customers/1/1/collectinfo-2015-09-03T230823-ns_1%4010.6.2.195.zip
      https://s3.amazonaws.com/cb-customers/1/1/collectinfo-2015-09-03T230823-ns_1%4010.6.2.233.zip
      https://s3.amazonaws.com/cb-customers/1/1/collectinfo-2015-09-03T230823-ns_1%4010.6.2.234.zip
      https://s3.amazonaws.com/cb-customers/1/1/collectinfo-2015-09-03T230823-ns_1%4010.6.2.237.zip
      https://s3.amazonaws.com/cb-customers/1/1/collectinfo-2015-09-03T230823-ns_1%4010.6.2.238.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Aliaksey Artamonau Aliaksey Artamonau (Inactive)
            ketaki Ketaki Gangal (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty