Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59734

[Rebalance] DELETE request for a collection during rebalance returns a 503 with error: {'Exceeded retries due to conflicting operations on bucket properties.'}

    XMLWordPrintable

Details

    Description

      Steps to reproduce

      1. Created a 6 node kv cluster

      2. Created 2 buckets

      3. Created 10 scopes per bucket and 50 collections per scope

      4. Loaded 5000 items onto each bucket (10 items per collection)

      5. Failed over node 172.23.121.71 and 172.23.96.168 gracefully

      6. Dropped collections during the failover

       

      Many DELETE requests return a 503

      DELETE https://172.23.96.196:18091/pools/default/buckets/bucket1/scopes/_default/collections/collection-12 body:  headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': u'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} 
       
      		 
      error: 503 reason: {u'_': u'Exceeded retries due to conflicting operations on bucket properties.'} {"errors":{"_":"Exceeded retries due to conflicting operations on bucket properties."}}

      Many such delete requests return the same 503

      $grep "503 87" ns_server.http_access.log
      		 
       
      		 
      172.23.123.70 - Administrator [20/Nov/2023:20:45:48 -0800] "DELETE /pools/default/buckets/bucket1/scopes/_default/collections/collection-12 HTTP/1.1" 503 87 - "python-requests/2.24.0" 3724
      		 
      172.23.123.70 - Administrator [20/Nov/2023:20:45:50 -0800] "DELETE /pools/default/buckets/bucket1/scopes/_default/collections/collection-47 HTTP/1.1" 503 87 - "python-requests/2.24.0" 5416
      		 
      172.23.123.70 - Administrator [20/Nov/2023:20:45:53 -0800] "DELETE /pools/default/buckets/bucket1/scopes/_default/collections/collection-14 HTTP/1.1" 503 87 - "python-requests/2.24.0" 7753

       

      Also noticing that these 503 responses are only observed with delete collection requests for ephemeral buckets

       

       

      TAF Script to reproduce

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i node.ini upgrade_version=7.6.0-1822 -t bucket_collections.collections_drop_recreate_rebalance.CollectionsDropRecreateRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,upgrade_version=7.6.0-1813,rerun=False,GROUP=set1,bucket_spec=multi_bucket.buckets_1000_collections,nodes_failover=2,get-cbcollect-info=True,log_level=debug,nodes_init=6,sirius_url=http://172.23.120.103:4000,infra_log_level=debug'

      Job name : debian-collections_drop_recreate_collections_rebalance_set1_7.0_P1

      Job link : http://cb-logs-qe.s3-website-us-west-2.amazonaws.com/7.6.0-1813/jenkins_logs/test_suite_executor-TAF/287320/

       

      Attachments

        Activity

          People

            raghav.sk Raghav S K
            raghav.sk Raghav S K
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty