Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-16264

Deleting bucket and removing nodes from the cluster, leaves the node unstable

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 4.5.0
    • 4.5.0
    • couchbase-bucket, memcached
    • Security Level: Public
    • 4.1.0-235 Enterprise Edition (build-235)

    Description

      1. Install CB Server on 4 nodes
      2. Create a bucket default
      3. Create a cluster of 4 nodes
      4. Add 1000 items to the cluster and verify the data is persisted.

      Now as part of the cleanup
      a. Bucket is deleted
      Get 500 error:
      2015-09-11 01:45:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'default'] on 172.23.105.131
      2015-09-11 01:45:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket default ...
      2015-09-11 01:45:39 | ERROR | MainProcess | test_thread | [rest_client._http_request] http://172.23.105.131:8091/pools/default/buckets/default error 500 reason: unknown

      {"_":"Bucket deletion not yet complete, but will continue.\r\n"}

      Seeing error:
      2015-09-11 01:46:45 | INFO | MainProcess | test_thread | [rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
      2015-09-11 01:46:45 | ERROR | MainProcess | test_thread | [rest_client._http_request] http://172.23.105.131:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
      Requested resource not found.
      2015-09-11 01:46:45 | ERROR | MainProcess | test_thread | [rest_client._http_request] http://172.23.105.131:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
      Requested resource not found.

      Rebalancing to remove all nodes in the cluster:
      2015-09-11 01:46:45 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%40172.23.105.182%2Cns_1%40172.23.105.210%2Cns_1%40172.23.105.132&user=Administrator&knownNodes=ns_1%40172.23.105.182%2Cns_1%40172.23.105.131%2Cns_1%40172.23.105.210%2Cns_1%40172.23.105.132
      2015-09-11 01:46:45 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started
      2015-09-11 01:46:45 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
      2015-09-11 01:46:55 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 10.0292150974 seconds
      2015-09-11 01:46:55 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance...
      2015-09-11 01:47:35 | ERROR | MainProcess | test_thread | [cluster_helper.cleanup_cluster] 'pools' on node 172.23.105.182:8091 - [

      {u'streamingUri': u'/poolsStreaming/default?uuid=05312a17f040add3c1a3c0f67bf5cf77', u'name': u'default', u'uri': u'/pools/default?uuid=05312a17f040add3c1a3c0f67bf5cf77'}

      ]
      2015-09-11 01:47:35 | ERROR | MainProcess | test_thread | [rest_client._http_request] http://172.23.105.210:8091/nodes/self error 404 reason: unknown "Node is unknown to this cluster."
      http://172.23.105.210:8091/nodes/self with status False: Node is unknown to this cluster.

      If you access the nodes in cluster, you will see that you can login to the node but all everything shows - Difficulties communicating with the cluster; displaying cached information

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jwalker Jim Walker
            ritam.sharma Ritam Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty