Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-23161

Bucket deletion timeout issue followed by rebalance failures found in XDCR tests

    XMLWordPrintable

Details

    • Triaged
    • Yes

    Description

      [2017-03-05 11:23:17,744] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.181:8091
      [2017-03-05 11:23:17,750] - [cluster_helper:80] INFO - ns_server @ 172.23.105.181:8091 is running
      Cluster instance shutdown with force
      [2017-03-05 11:23:17,750] - [xdcrnewbasetests:1149] INFO - removing xdcr/nodes settings
      [2017-03-05 11:23:17,813] - [rest_client:1025] INFO - removing remote cluster name:remote_cluster_C2-C1
      [2017-03-05 11:23:17,848] - [xdcrnewbasetests:1159] INFO - cleanup [ip:172.23.107.120 port:8091 ssh_username:root, ip:172.23.107.169 port:8091 ssh_username:root, ip:172.23.105.180 port:8091 ssh_username:root]
      [2017-03-05 11:23:17,886] - [bucket_helper:138] INFO - deleting existing buckets [u'default'] on 172.23.107.120
      [2017-03-05 11:23:17,886] - [bucket_helper:140] INFO - remove bucket default ...
      [2017-03-05 11:23:47,906] - [rest_client:790] ERROR - http://172.23.107.120:8091/pools/default/buckets/default error 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"}
      [2017-03-05 11:23:47,906] - [rest_client:1898] WARNING - Bucket deletion timed out waiting for all nodes
      [2017-03-05 11:23:47,931] - [bucket_helper:153] ERROR - Unable to get timings for bucket
      [2017-03-05 11:23:47,931] - [bucket_helper:154] INFO - deleted bucket : default from 172.23.107.120
      [2017-03-05 11:23:47,931] - [bucket_helper:230] INFO - waiting for bucket deletion to complete....
      [2017-03-05 11:23:47,955] - [rest_client:134] INFO - node 172.23.107.120 existing buckets : []
      [2017-03-05 11:23:48,005] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes
      [2017-03-05 11:23:48,010] - [rest_client:1322] INFO - rebalance params : password=welcome&ejectedNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.105.180&user=Administrator&knownNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.105.180%2Cns_1%40172.23.107.120
      [2017-03-05 11:23:48,017] - [rest_client:1326] INFO - rebalance operation started
      [2017-03-05 11:23:48,047] - [rest_client:1457] INFO - rebalance percentage : 0.00 %
      [2017-03-05 11:23:58,064] - [rest_client:1457] INFO - rebalance percentage : 0.00 %
      [2017-03-05 11:24:08,083] - [rest_client:1440] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
      [2017-03-05 11:24:08,208] - [rest_client:2726] INFO - Latest logs from UI on 172.23.107.120:
      [2017-03-05 11:24:08,208] - [rest_client:2727] ERROR - {u'node': u'ns_1@172.23.107.120', u'code': 0
      

      Bucket deletion issue found in XDCR test cleanup - same as MB-16745 - attaching logs and console output

      Attachments

        1. 172.23.105.102.zip
          2.67 MB
        2. 172.23.105.178-20170305-1116-diag.zip
          4.92 MB
        3. 172.23.105.179-20170305-1117-diag.zip
          913 kB
        4. 172.23.105.180-20170305-1118-diag.zip
          3.63 MB
        5. 172.23.105.181-20170305-1120-diag.zip
          4.67 MB
        6. 172.23.105.20.zip
          6.53 MB
        7. 172.23.105.3.zip
          6.15 MB
        8. 172.23.105.4.zip
          4.24 MB
        9. 172.23.105.67.zip
          2.64 MB
        10. 172.23.107.120-20170305-1111-diag.zip
          8.73 MB
        11. 172.23.107.169-20170305-1114-diag.zip
          5.69 MB
        12. 172.23.107.171-20170305-1109-diag.zip
          6.68 MB
        13. 172.23.107.198-20170305-1106-diag.zip
          10.12 MB
        14. 172.23.107.53.zip
          2.27 MB
        15. consoleFull.rtf
          91 kB

        Issue Links

          Activity

            People

              yu Yu Sui (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty