Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19767

Issue with bucket deletion with XDCR tests

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 4.6.0, 5.0.0
    • 4.5.0
    • memcached, XDCR
    • None
    • 2594

    Description

      JOB:
      http://qa.hq.northscale.net/job/cen006-p0-xxdcr-vset09-00-goxdcr-pause-resume/222/consoleFull

      TC - test_6:

      ./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,fail_on_errors=1,GROUP=P0 -t xdcr.pauseResumeXDCR.PauseResumeTest.replication_with_pause_and_resume,items=80000,consecutive_pause_resume=3,rdirection=bidirection,ctopology=chain,standard_buckets=1,sasl_buckets=1,replication_type=xmem,pause=C1,update=C1-C2,delete=C1-C2,GROUP=P0;xmim
      

      ERROR:

      [2016-05-27 02:47:06,958] - [bucket_helper:139] INFO - remove bucket default ...
      [2016-05-27 02:47:36,984] - [rest_client:778] ERROR - http://10.5.2.228:8091/pools/default/buckets/default error 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"}
      [2016-05-27 02:47:36,985] - [rest_client:1803] WARNING - Bucket deletion timed out waiting for all nodes
      [2016-05-27 02:47:37,013] - [bucket_helper:152] ERROR - Unable to get timings for bucket
      [2016-05-27 02:47:37,013] - [bucket_helper:153] INFO - deleted bucket : default from 10.5.2.228
      [2016-05-27 02:47:37,014] - [bucket_helper:229] INFO - waiting for bucket deletion to complete....
      [2016-05-27 02:47:37,083] - [rest_client:133] INFO - node 10.5.2.228 existing buckets : [u'sasl_bucket_1', u'standard_bucket_1']
      [2016-05-27 02:47:37,085] - [bucket_helper:139] INFO - remove bucket sasl_bucket_1 ...
      [2016-05-27 02:48:07,101] - [rest_client:778] ERROR - http://10.5.2.228:8091/pools/default/buckets/sasl_bucket_1 error 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"}
      [2016-05-27 02:48:07,101] - [rest_client:1803] WARNING - Bucket deletion timed out waiting for all nodes
      [2016-05-27 02:48:07,121] - [bucket_helper:152] ERROR - Unable to get timings for bucket
      [2016-05-27 02:48:07,121] - [bucket_helper:153] INFO - deleted bucket : sasl_bucket_1 from 10.5.2.228
      [2016-05-27 02:48:07,121] - [bucket_helper:229] INFO - waiting for bucket deletion to complete....
      [2016-05-27 02:48:07,146] - [rest_client:133] INFO - node 10.5.2.228 existing buckets : [u'standard_bucket_1']
      [2016-05-27 02:48:07,147] - [bucket_helper:139] INFO - remove bucket standard_bucket_1 ...
      [2016-05-27 02:48:37,159] - [rest_client:778] ERROR - http://10.5.2.228:8091/pools/default/buckets/standard_bucket_1 error 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"}
      [2016-05-27 02:48:37,160] - [rest_client:1803] WARNING - Bucket deletion timed out waiting for all nodes
      [2016-05-27 02:48:37,174] - [bucket_helper:152] ERROR - Unable to get timings for bucket
      [2016-05-27 02:48:37,174] - [bucket_helper:153] INFO - deleted bucket : standard_bucket_1 from 10.5.2.228
      [2016-05-27 02:48:37,175] - [bucket_helper:229] INFO - waiting for bucket deletion to complete....
      [2016-05-27 02:48:37,180] - [rest_client:133] INFO - node 10.5.2.228 existing buckets : []
      [2016-05-27 02:48:37,211] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes
      [2016-05-27 02:48:37,217] - [rest_client:1283] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.5.2.229&user=Administrator&knownNodes=ns_1%4010.5.2.229%2Cns_1%4010.5.2.228
      [2016-05-27 02:48:37,229] - [rest_client:1287] INFO - rebalance operation started
      [2016-05-27 02:48:37,248] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
      [2016-05-27 02:48:47,264] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
      [2016-05-27 02:48:57,280] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
      [2016-05-27 02:49:07,296] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
      [2016-05-27 02:49:17,312] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
      [2016-05-27 02:49:27,328] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
      [2016-05-27 02:49:37,343] - [rest_client:1401] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
      [2016-05-27 02:49:37,776] - [rest_client:2529] INFO - Latest logs from UI on 10.5.2.228:
      [2016-05-27 02:49:37,776] - [rest_client:2530] ERROR - {u'node': u'ns_1@10.5.2.228', u'code': 2, u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n                              [{\'ns_1@10.5.2.228\',\n                                {\'EXIT\',\n                                 {old_buckets_shutdown_wait_failed,\n                                  ["default","standard_bucket_1",\n                                   "sasl_bucket_1"]}}}]}\n', u'shortText': u'message', u'serverTime': u'2016-05-27T02:49:37.224Z', u'module': u'ns_orchestrator', u'tstamp': 1464342577224, u'type': u'info'}
      [2016-05-27 02:49:37,776] - [rest_client:2530] ERROR - {u'node': u'ns_1@10.5.2.228', u'code': 0, u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@10.5.2.228\',\n                                                         {\'EXIT\',\n                                                          {old_buckets_shutdown_wait_failed,\n                                                           ["default",\n                                                            "standard_bucket_1",\n                                                            "sasl_bucket_1"]}}}]\n', u'shortText': u'message', u'serverTime': u'2016-05-27T02:49:37.224Z', u'module': u'ns_rebalancer', u'tstamp': 1464342577224, u'type': u'critical'}
      [2016-05-27 02:49:37,777] - [rest_client:2530] ERROR - {u'node': u'ns_1@10.5.2.228', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.5.2.228'], EjectNodes = ['ns_1@10.5.2.229'], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2016-05-27T02:48:37.213Z', u'module': u'ns_orchestrator', u'tstamp': 1464342517213, u'type': u'info'}
      [2016-05-27 02:49:37,777] - [rest_client:2530] ERROR - {u'node': u'ns_1@10.5.2.228', u'code': 102, u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@10.5.2.228\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@10.5.2.229\']\n', u'shortText': u'client-side error report', u'serverTime': u'2016-05-27T02:48:37.205Z', u'module': u'menelaus_web', u'tstamp': 1464342517205, u'type': u'warning'}
      [2016-05-27 02:49:37,777] - [rest_client:2530] ERROR - {u'node': u'ns_1@10.5.2.229', u'code': 0, u'text': u'Shutting down bucket "sasl_bucket_1" on \'ns_1@10.5.2.229\' for deletion', u'shortText': u'message', u'serverTime': u'2016-05-27T02:47:38.422Z', u'module': u'ns_memcached', u'tstamp': 1464342458422, u'type': u'info'}
      [2016-05-27 02:49:37,777] - [rest_client:2530] ERROR - {u'node': u'ns_1@10.5.2.228', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@10.5.2.228\' for deletion', u'shortText': u'message', u'serverTime': u'2016-05-27T02:47:08.534Z', u'module': u'ns_memcached', u'tstamp': 1464342428534, u'type': u'info'}
      [2016-05-27 02:49:37,778] - [rest_client:2530] ERROR - {u'node': u'ns_1@10.5.2.229', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@10.5.2.229\' for deletion', u'shortText': u'message', u'serverTime': u'2016-05-27T02:47:08.331Z', u'module': u'ns_memcached', u'tstamp': 1464342428331, u'type': u'info'}
      [2016-05-27 02:49:37,778] - [rest_client:2530] ERROR - {u'node': u'ns_1@10.5.2.228', u'code': 0, u'text': u'Remote cluster reference "remote_cluster_C1-C2" known via 10.5.2.230:8091 removed.', u'shortText': u'message', u'serverTime': u'2016-05-27T02:47:06.820Z', u'module': u'xdcr', u'tstamp': 1464342426820, u'type': u'info'}
      [2016-05-27 02:49:37,778] - [rest_client:2530] ERROR - {u'node': u'ns_1@10.5.2.228', u'code': 0, u'text': u'Replication from bucket "standard_bucket_1" to bucket "standard_bucket_1" on cluster "remote_cluster_C1-C2" removed.', u'shortText': u'message', u'serverTime': u'2016-05-27T02:47:06.738Z', u'module': u'xdcr', u'tstamp': 1464342426738, u'type': u'info'}
      [2016-05-27 02:49:37,779] - [rest_client:2530] ERROR - {u'node': u'ns_1@10.5.2.228', u'code': 0, u'text': u'Replication from bucket "sasl_bucket_1" to bucket "sasl_bucket_1" on cluster "remote_cluster_C1-C2" removed.', u'shortText': u'message', u'serverTime': u'2016-05-27T02:47:06.706Z', u'module': u'xdcr', u'tstamp': 1464342426706, u'type': u'info'}
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              ritam.sharma Ritam Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty