Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45166

[Collections] error,exceeded_retries during initial bulk creation of 1000 collections

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 7.0.0
    • Cheshire-Cat
    • ns_server
    • Centos 7 64 bit; Couchbase EE 4744

    Description

      Summary:
      Initial creation of scopes and collections of roughly 1000 collections on 15 buckets via bulk api failed (during one of the collection creations)

      Steps to Repro
      1. Create a 6 node cluster

      2021-03-22 21:08:33,003 | test  | INFO    | pool-1-thread-16 | [table_view:display:72] Rebalance Overview
      +----------------+--------------+-----------------------+----------------+--------------+
      | Nodes          | Services     | Version               | CPU            | Status       |
      +----------------+--------------+-----------------------+----------------+--------------+
      | 172.23.105.175 | kv           | 7.0.0-4744-enterprise | 0.337837837838 | Cluster node |
      | 172.23.106.233 | ['n1ql']     |                       |                | <--- IN ---  |
      | 172.23.106.236 | ['n1ql']     |                       |                | <--- IN ---  |
      | 172.23.106.238 | ['kv,index'] |                       |                | <--- IN ---  |
      | 172.23.106.250 | ['kv,index'] |                       |                | <--- IN ---  |
      | 172.23.106.251 | ['kv,index'] |                       |                | <--- IN ---  |
      +----------------+--------------+-----------------------+----------------+--------------+
      

      2. create 15 buckets and use bulk api to create 66 scopes, 1 collection each (so a total of 990 scopes and collections)
      failed midway with unexpected server error

      on .175 error.log
      I see a couple of exceeded retries error

      [ns_server:error,2021-03-22T21:08:49.120-07:00,ns_1@172.23.105.175:<0.19533.0>:menelaus_util:reply_server_error:208]Server error during processing: ["web request failed",
                                       {path,
                                        "/pools/default/buckets/f7RvDBq7Etg3BJbnH60x9RcBEN5DRal2koJusKR0Og--48-495000/scopes"},
                                       {method,'PUT'},
                                       {type,exit},
                                       {what,
                                        {{{case_clause,{error,exceeded_retries}},

      on .175 debug.log

      =========================CRASH REPORT=========================
        crasher:
          initial call: work_queue:init/1
          pid: <0.16648.0>
          registered_name: []
          exception error: no case clause matching {error,exceeded_retries}
            in function  collections:do_update/2 (src/collections.erl, line 276)
            in call from work_queue:handle_call/3 (src/work_queue.erl, line 57)
            in call from gen_server:try_handle_call/4 (gen_server.erl, line 661)
            in call from gen_server:handle_msg/6 (gen_server.erl, line 690)
          ancestors: [mb_master_sup,mb_master,leader_registry_sup,
                        leader_services_sup,<0.628.0>,ns_server_sup,
                        ns_server_nodes_sup,<0.253.0>,ns_server_cluster_sup,
                        root_sup,<0.138.0>]
          message_queue_len: 11

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sumedh.basarkod Sumedh Basarkod (Inactive)
            sumedh.basarkod Sumedh Basarkod (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty