Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6968

bucket deletion was slow due to apparently slow master vbucket deletion (was: Rebalance out failed with reason {buckets_shutdown_wait_failed)

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.0
    • Fix Version/s: 2.0.1
    • Component/s: ns_server, view-engine
    • Security Level: Public
    • Labels:
      None
    • Environment:

      Description

      -t viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,max-dupe-result-count=10,num_nodes_to_add=3,skip_rebalance=true,docs-per-day=1000

      Rebalance out 4->3 is failed:

      [rebalance:error,2012-10-18T19:39:33.308,ns_1@10.3.3.58:<0.26558.105>:ns_rebalancer:do_wait_buckets_shutdown:204]Failed to wait deletion of some buckets on some nodes: [{'ns_1@10.3.3.58',
      {'EXIT',

      {old_buckets_shutdown_wait_failed, ["default"]}

      }}]

      [user:info,2012-10-18T19:39:33.311,ns_1@10.3.3.58:<0.4365.96>:ns_orchestrator:handle_info:319]Rebalance exited with reason {buckets_shutdown_wait_failed,
      [{'ns_1@10.3.3.58',
      {'EXIT',

      {old_buckets_shutdown_wait_failed, ["default"]}

      }}]}
      logs:
      http://qa.hq.northscale.net/job/centos-64-2.0-views-regression/29/artifact/logs/testrunner-12-Oct-17_23-58-20/483c326b-c52d-4555-8fc2-482a16dca098-10.3.3.58-diag.txt.gz
      http://qa.hq.northscale.net/job/centos-64-2.0-views-regression/29/artifact/logs/testrunner-12-Oct-17_23-58-20/483c326b-c52d-4555-8fc2-482a16dca098-10.3.3.64-diag.txt.gz
      http://qa.hq.northscale.net/job/centos-64-2.0-views-regression/29/artifact/logs/testrunner-12-Oct-17_23-58-20/483c326b-c52d-4555-8fc2-482a16dca098-10.3.3.68-diag.txt.gz
      http://qa.hq.northscale.net/job/centos-64-2.0-views-regression/29/artifact/logs/testrunner-12-Oct-17_23-58-20/483c326b-c52d-4555-8fc2-482a16dca098-10.3.3.71-diag.txt.gz

      1. 10.1.3.135-8091-diag.txt.gz
        86 kB
        Iryna
      2. 10.1.3.146-8091-diag.txt.gz
        852 kB
        Iryna
      3. 10.1.3.147-8091-diag.txt.gz
        104 kB
        Iryna
      4. 10.1.3.151-8091-diag.txt.gz
        58 kB
        Iryna
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        steve Steve Yen added a comment -

        moving to 2.0.1 per bug-scrub mtg

        Show
        steve Steve Yen added a comment - moving to 2.0.1 per bug-scrub mtg
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Just to update. When I mentioned master vbucket in title, I was actually referring to couch db file master.couch where design docs are stored. Apparently when it gets deleted view engine gets rid of index files and that was slow in this particular case.

        Would be nice if QE can reproduce it and attach new set of logs in case it'll provide us more info.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Just to update. When I mentioned master vbucket in title, I was actually referring to couch db file master.couch where design docs are stored. Apparently when it gets deleted view engine gets rid of index files and that was slow in this particular case. Would be nice if QE can reproduce it and attach new set of logs in case it'll provide us more info.
        Hide
        steve Steve Yen added a comment -

        per-bug-scrub, assign back to Iryna to see if it's reproducible.

        Show
        steve Steve Yen added a comment - per-bug-scrub, assign back to Iryna to see if it's reproducible.
        Hide
        iryna iryna added a comment -

        tested build 1967
        not reproduced

        Show
        iryna iryna added a comment - tested build 1967 not reproduced
        Hide
        iryna iryna added a comment -

        tested with 2.0.1, closed

        Show
        iryna iryna added a comment - tested with 2.0.1, closed

          People

          • Assignee:
            iryna iryna
            Reporter:
            iryna iryna
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes