Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40654

[Collections] scope/collection creation/drop fails with 503 status error

    XMLWordPrintable

Details

    Description

      Summary:
      Some Scope/collection creation/drop fails with 

      "error: 503 reason: unknown "Operation is unsafe at this time. Retry later."
      Unsure if this is something expected as with some sleep put (or retries later) to ensure these drops/creation happen after rebalance has started progress, it doesn't throw this error. 

      Steps to Reproduce

      1. Create a 3 node cluster
        2020-07-28 06:01:59,322 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview
        -----------------------++-------------
        Nodes Services Status

      -----------------------++-------------

      172.23.105.211 kv Cluster node
      172.23.105.212 None <--- IN —
      172.23.105.213 None <--- IN —

      -----------------------++-------------

           2. Create buckets with collections and load data
      2020-07-28 06:12:05,461 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
      -----------------+++-------------------------------+---------------------------------

      Bucket Type Replicas Durability TTL Items RAM Quota RAM Used Disk Used

      -----------------+++-------------------------------+---------------------------------

      bucket1 couchbase 3 none 0 3000 629145600 184798896 281126164
      bucket2 ephemeral 3 none 0 3000 629145600 266479728 102
      default couchbase 3 none 0 500000 6291456000 511031552 455611628

      -----------------+++-------------------------------+---------------------------------

         3.   Start rebalance-in op with CRUD on collections in parallel
      Some CRUD on collections fail. Taking one such example:

      2020-07-28 06:12:08,174 | infra | ERROR | Thread-1839 | [Rest_Connection:_http_request:205] DELETE http://172.23.105.211:8091/pools/default/buckets/bucket2/collections/_default/mA body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 503 reason: unknown "Operation is unsafe at this time. Retry later." auth: Administrator:password
      2020-07-28 06:12:08,181 | test | ERROR | Thread-1839 | [bucket_ready_functions:drop_collection:814] Collection 'bucket2:_default:mA' delete failed: "Operation is unsafe at this time. Retry later."

      Rebalance starts progress at:
      2020-07-28 06:12:12,269 | test | INFO | pool-2-thread-20 | [table_view:display:72] Rebalance Overview
      -----------------------++-------------

      Nodes Services Status

      -----------------------++-------------

      172.23.105.212 kv Cluster node
      172.23.105.213 kv Cluster node
      172.23.105.211 kv Cluster node
      172.23.105.215 None <--- IN —
      172.23.105.217 None <--- IN —

      -----------------------++-------------
      Have attached the consoleText. Screenshot of UI showing collection mA still present. Have attached the cb-collect-info.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            artem Artem Stemkovski
            sumedh.basarkod Sumedh Basarkod (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty