Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-23161

Bucket deletion timeout issue followed by rebalance failures found in XDCR tests

    XMLWordPrintable

Details

    • Triaged
    • Yes

    Description

      [2017-03-05 11:23:17,744] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.181:8091
      [2017-03-05 11:23:17,750] - [cluster_helper:80] INFO - ns_server @ 172.23.105.181:8091 is running
      Cluster instance shutdown with force
      [2017-03-05 11:23:17,750] - [xdcrnewbasetests:1149] INFO - removing xdcr/nodes settings
      [2017-03-05 11:23:17,813] - [rest_client:1025] INFO - removing remote cluster name:remote_cluster_C2-C1
      [2017-03-05 11:23:17,848] - [xdcrnewbasetests:1159] INFO - cleanup [ip:172.23.107.120 port:8091 ssh_username:root, ip:172.23.107.169 port:8091 ssh_username:root, ip:172.23.105.180 port:8091 ssh_username:root]
      [2017-03-05 11:23:17,886] - [bucket_helper:138] INFO - deleting existing buckets [u'default'] on 172.23.107.120
      [2017-03-05 11:23:17,886] - [bucket_helper:140] INFO - remove bucket default ...
      [2017-03-05 11:23:47,906] - [rest_client:790] ERROR - http://172.23.107.120:8091/pools/default/buckets/default error 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"}
      [2017-03-05 11:23:47,906] - [rest_client:1898] WARNING - Bucket deletion timed out waiting for all nodes
      [2017-03-05 11:23:47,931] - [bucket_helper:153] ERROR - Unable to get timings for bucket
      [2017-03-05 11:23:47,931] - [bucket_helper:154] INFO - deleted bucket : default from 172.23.107.120
      [2017-03-05 11:23:47,931] - [bucket_helper:230] INFO - waiting for bucket deletion to complete....
      [2017-03-05 11:23:47,955] - [rest_client:134] INFO - node 172.23.107.120 existing buckets : []
      [2017-03-05 11:23:48,005] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes
      [2017-03-05 11:23:48,010] - [rest_client:1322] INFO - rebalance params : password=welcome&ejectedNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.105.180&user=Administrator&knownNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.105.180%2Cns_1%40172.23.107.120
      [2017-03-05 11:23:48,017] - [rest_client:1326] INFO - rebalance operation started
      [2017-03-05 11:23:48,047] - [rest_client:1457] INFO - rebalance percentage : 0.00 %
      [2017-03-05 11:23:58,064] - [rest_client:1457] INFO - rebalance percentage : 0.00 %
      [2017-03-05 11:24:08,083] - [rest_client:1440] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
      [2017-03-05 11:24:08,208] - [rest_client:2726] INFO - Latest logs from UI on 172.23.107.120:
      [2017-03-05 11:24:08,208] - [rest_client:2727] ERROR - {u'node': u'ns_1@172.23.107.120', u'code': 0
      

      Bucket deletion issue found in XDCR test cleanup - same as MB-16745 - attaching logs and console output

      Attachments

        1. consoleFull.rtf
          91 kB
        2. 172.23.105.180-20170305-1118-diag.zip
          3.63 MB
        3. 172.23.107.169-20170305-1114-diag.zip
          5.69 MB
        4. 172.23.105.181-20170305-1120-diag.zip
          4.67 MB
        5. 172.23.105.179-20170305-1117-diag.zip
          913 kB
        6. 172.23.107.171-20170305-1109-diag.zip
          6.68 MB
        7. 172.23.107.198-20170305-1106-diag.zip
          10.12 MB
        8. 172.23.105.178-20170305-1116-diag.zip
          4.92 MB
        9. 172.23.107.120-20170305-1111-diag.zip
          8.73 MB
        10. 172.23.107.53.zip
          2.27 MB
        11. 172.23.105.102.zip
          2.67 MB
        12. 172.23.105.4.zip
          4.24 MB
        13. 172.23.105.67.zip
          2.64 MB
        14. 172.23.105.3.zip
          6.15 MB
        15. 172.23.105.20.zip
          6.53 MB

        Issue Links

          Activity

            People

              yu Yu Sui (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty