Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6711

size of one vBucket is 10 GB - Rebalance exited with reason replicator_died( many retries to notify CouchDB of update : vbid=294 rev=1)

    Details

      Description

      2.0.0-1751-rel

      testrunner -i /tmp/rebalance_regression.ini get-logs=True,disabled_consistent_view=False -t swaprebalance.SwapRebalanceFailedTests.test_failed_swap_rebalance,replica=2,num-buckets=1,num-swap=2,swap-orchestrator=False
      http://qa.hq.northscale.net/job/centos-64-2.0-rebalance-regressions/47/consoleFull

      2012-09-21 06:26:25.276 menelaus_web_alerts_srv:1:info:message(ns_1@10.3.121.94) - Approaching full disk warning. Usage of disk "/" on node "10.3.121.94" is around 91%.
      2012-09-21 06:43:15.462 supervisor_cushion:1:warning:port exited too soon after restart(ns_1@10.3.121.94) - Service memcached exited on node 'ns_1@10.3.121.94' in 0.58s

      2012-09-21 06:44:30.881 ns_vbucket_mover:0:critical:message(ns_1@10.3.121.92) - <0.17933.33> exited with {exited,
      {'EXIT',<0.17965.33>,
      {replicator_died,
      {'EXIT',<19812.318.4>,downstream_closed}}}}
      2012-09-21 06:44:30.892 ns_memcached:2:info:message(ns_1@10.3.121.96) - Shutting down bucket "bucket-0" on 'ns_1@10.3.121.96' for deletion
      2012-09-21 06:44:31.274 ns_orchestrator:2:info:message(ns_1@10.3.121.92) - Rebalance exited with reason {exited,
      {'EXIT',<0.17965.33>,
      {replicator_died,
      {'EXIT',<19812.318.4>,downstream_closed}}}}

      logs on vm 10.3.121.94 contain many retries to notify CouchDB of update : vbid=294 rev=1

      memcached<0.31491.3>: Fri Sep 21 06:12:14.929498 PDT 3: Retry notify CouchDB of update, vbid=294 rev=1
      memcached<0.31491.3>: Fri Sep 21 06:12:14.931793 PDT 3: Retry notify CouchDB of update, vbid=294 rev=1
      root@ubuntu1104-64:/opt/couchbase/var/lib/couchbase/logs# more info.5
      [ns_server:info,2012-09-21T6:06:34.551,ns_1@10.3.121.94:ns_port_memcached:ns_port_server:log:169]memcached<0.31491.3>: Fri Sep 21 06:06:34.3
      50714 PDT 3: Retry notify CouchDB of update, vbid=294 rev=1
      memcached<0.31491.3>: Fri Sep 21 06:06:34.352976 PDT 3: Retry notify CouchDB of update, vbid=294 rev=1
      memcached<0.31491.3>: Fri Sep 21 06:06:34.355496 PDT 3: Retry notify CouchDB of update, vbid=294 rev=1

      as a result the size of vbucket 294 is 9.3 GB?

      root@ubuntu1104-64:/opt/couchbase/var/lib/couchbase/data/bucket-2# ls -la
      total 9337196
      drwxr-xr-x 2 couchbase couchbase 4096 2012-09-21 06:44 .
      drwxr-xr-x 4 couchbase couchbase 4096 2012-09-21 03:54 ..
      rw-rr- 1 couchbase couchbase 442459 2012-09-21 04:11 0.couch.16
      rw-rr- 1 couchbase couchbase 397403 2012-09-21 04:11 147.couch.14
      rw-rr- 1 couchbase couchbase 438363 2012-09-21 06:44 148.couch.14
      rw-rr- 1 couchbase couchbase 413787 2012-09-21 04:11 149.couch.14
      rw-rr- 1 couchbase couchbase 417883 2012-09-21 06:44 150.couch.14
      rw-rr- 1 couchbase couchbase 454747 2012-09-21 06:44 151.couch.13
      rw-rr- 1 couchbase couchbase 442459 2012-09-21 04:11 152.couch.13
      rw-rr- 1 couchbase couchbase 430171 2012-09-21 04:11 153.couch.13
      rw-rr- 1 couchbase couchbase 413787 2012-09-21 06:44 154.couch.13
      rw-rr- 1 couchbase couchbase 409691 2012-09-21 04:11 172.couch.12
      rw-rr- 1 couchbase couchbase 413787 2012-09-21 04:11 173.couch.12
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 174.couch.12
      rw-rr- 1 couchbase couchbase 405595 2012-09-21 04:11 175.couch.12
      rw-rr- 1 couchbase couchbase 397403 2012-09-21 04:11 176.couch.12
      rw-rr- 1 couchbase couchbase 438363 2012-09-21 04:11 177.couch.12
      rw-rr- 1 couchbase couchbase 405595 2012-09-21 04:11 178.couch.11
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 179.couch.11
      rw-rr- 1 couchbase couchbase 413787 2012-09-21 04:11 1.couch.16
      rw-rr- 1 couchbase couchbase 1355867 2012-09-21 03:54 25.couch.1
      rw-rr- 1 couchbase couchbase 9494917305 2012-09-21 06:43 294.couch.1
      rw-rr- 1 couchbase couchbase 430171 2012-09-21 04:11 294.couch.11
      rw-rr- 1 couchbase couchbase 421979 2012-09-21 04:11 295.couch.11
      rw-rr- 1 couchbase couchbase 405595 2012-09-21 04:11 296.couch.11
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 297.couch.10
      rw-rr- 1 couchbase couchbase 409691 2012-09-21 04:11 298.couch.10
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 299.couch.10
      rw-rr- 1 couchbase couchbase 389211 2012-09-21 04:11 2.couch.16
      rw-rr- 1 couchbase couchbase 421979 2012-09-21 04:11 300.couch.10
      rw-rr- 1 couchbase couchbase 405595 2012-09-21 04:11 301.couch.9
      rw-rr- 1 couchbase couchbase 594011 2012-09-21 03:54 33.couch.1
      rw-rr- 1 couchbase couchbase 430171 2012-09-21 04:11 343.couch.9
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 344.couch.9
      rw-rr- 1 couchbase couchbase 438363 2012-09-21 04:11 345.couch.8
      rw-rr- 1 couchbase couchbase 454747 2012-09-21 04:11 346.couch.8
      rw-rr- 1 couchbase couchbase 405595 2012-09-21 04:11 347.couch.8
      rw-rr- 1 couchbase couchbase 401499 2012-09-21 04:11 348.couch.8
      rw-rr- 1 couchbase couchbase 421979 2012-09-21 04:11 349.couch.7
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 350.couch.7
      rw-rr- 1 couchbase couchbase 430171 2012-09-21 04:11 351.couch.7
      rw-rr- 1 couchbase couchbase 409691 2012-09-21 04:11 352.couch.7
      rw-rr- 1 couchbase couchbase 405595 2012-09-21 04:11 353.couch.7
      rw-rr- 1 couchbase couchbase 401499 2012-09-21 04:11 354.couch.6
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 355.couch.6
      rw-rr- 1 couchbase couchbase 397403 2012-09-21 04:11 356.couch.6
      rw-rr- 1 couchbase couchbase 442459 2012-09-21 04:11 357.couch.5
      rw-rr- 1 couchbase couchbase 438363 2012-09-21 04:11 358.couch.5
      rw-rr- 1 couchbase couchbase 450651 2012-09-21 04:11 359.couch.5
      rw-rr- 1 couchbase couchbase 421979 2012-09-21 04:11 360.couch.5
      rw-rr- 1 couchbase couchbase 405595 2012-09-21 04:11 361.couch.5
      rw-rr- 1 couchbase couchbase 421979 2012-09-21 04:11 362.couch.5
      rw-rr- 1 couchbase couchbase 430171 2012-09-21 04:11 363.couch.4
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 364.couch.4
      rw-rr- 1 couchbase couchbase 430171 2012-09-21 04:11 365.couch.4
      rw-rr- 1 couchbase couchbase 450651 2012-09-21 04:11 366.couch.4
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 367.couch.4
      rw-rr- 1 couchbase couchbase 430171 2012-09-21 04:11 392.couch.4
      rw-rr- 1 couchbase couchbase 413787 2012-09-21 04:11 393.couch.4
      rw-rr- 1 couchbase couchbase 413787 2012-09-21 04:11 394.couch.4
      rw-rr- 1 couchbase couchbase 413787 2012-09-21 04:11 395.couch.4
      rw-rr- 1 couchbase couchbase 442459 2012-09-21 04:11 396.couch.4
      rw-rr- 1 couchbase couchbase 417883 2012-09-21 04:11 397.couch.4
      rw-rr- 1 couchbase couchbase 430171 2012-09-21 04:11 398.couch.4
      rw-rr- 1 couchbase couchbase 23777465 2012-09-21 06:44 399.couch.1
      rw-rr- 1 couchbase couchbase 421979 2012-09-21 04:11 399.couch.4
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 3.couch.16
      rw-rr- 1 couchbase couchbase 999515 2012-09-21 03:54 42.couch.1
      rw-rr- 1 couchbase couchbase 450651 2012-09-21 04:11 4.couch.15
      rw-rr- 1 couchbase couchbase 442459 2012-09-21 04:11 594.couch.3
      rw-rr- 1 couchbase couchbase 450651 2012-09-21 04:11 595.couch.3
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 596.couch.3
      rw-rr- 1 couchbase couchbase 401499 2012-09-21 04:11 597.couch.3
      rw-rr- 1 couchbase couchbase 430171 2012-09-21 04:11 598.couch.3
      rw-rr- 1 couchbase couchbase 421979 2012-09-21 04:11 599.couch.3
      rw-rr- 1 couchbase couchbase 401499 2012-09-21 04:11 5.couch.15
      rw-rr- 1 couchbase couchbase 454747 2012-09-21 04:11 600.couch.3
      rw-rr- 1 couchbase couchbase 413787 2012-09-21 04:11 601.couch.2
      rw-rr- 1 couchbase couchbase 409691 2012-09-21 04:11 634.couch.2
      rw-rr- 1 couchbase couchbase 417883 2012-09-21 04:11 635.couch.2
      rw-rr- 1 couchbase couchbase 405595 2012-09-21 04:11 636.couch.2
      rw-rr- 1 couchbase couchbase 442459 2012-09-21 04:11 637.couch.2
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 638.couch.2
      rw-rr- 1 couchbase couchbase 421979 2012-09-21 04:11 639.couch.2
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 640.couch.2
      rw-rr- 1 couchbase couchbase 434267 2012-09-21 04:11 641.couch.2
      rw-rr- 1 couchbase couchbase 413787 2012-09-21 04:11 642.couch.2
      rw-rr- 1 couchbase couchbase 430171 2012-09-21 04:11 643.couch.2
      rw-rr- 1 couchbase couchbase 434267 2012-09-21 04:11 644.couch.2
      rw-rr- 1 couchbase couchbase 401499 2012-09-21 04:11 645.couch.2
      rw-rr- 1 couchbase couchbase 438363 2012-09-21 04:11 646.couch.2
      rw-rr- 1 couchbase couchbase 446555 2012-09-21 04:11 647.couch.2
      rw-rr- 1 couchbase couchbase 413787 2012-09-21 04:11 648.couch.2
      rw-rr- 1 couchbase couchbase 446555 2012-09-21 04:11 649.couch.2
      rw-rr- 1 couchbase couchbase 417883 2012-09-21 04:11 6.couch.15
      rw-rr- 1 couchbase couchbase 426075 2012-09-21 04:11 7.couch.15
      rw-rr- 1 couchbase couchbase 417883 2012-09-21 04:11 8.couch.15
      rw-rr- 1 couchbase couchbase 4130 2012-09-21 04:11 master.couch.16
      rw-rr- 1 couchbase couchbase 18915 2012-09-21 04:29 stats.json
      rw-rr- 1 couchbase couchbase 0 2012-09-21 06:44 stats.json.new
      rw-rr- 1 couchbase couchbase 18915 2012-09-21 04:28 stats.json.old
      root@ubuntu1104-64:/opt/couchbase/var/lib/couchbase/data/bucket-2# ^C

      1. 3a0ad78b-cf67-4e70-8d24-31eba49e4a41-10.3.121.92-diag.txt.gz
        6.74 MB
        Andrei Baranouski
      2. 3a0ad78b-cf67-4e70-8d24-31eba49e4a41-10.3.121.93-diag.txt.gz
        8.99 MB
        Andrei Baranouski
      3. 3a0ad78b-cf67-4e70-8d24-31eba49e4a41-10.3.121.95-diag.txt.gz
        7.90 MB
        Andrei Baranouski
      4. 3a0ad78b-cf67-4e70-8d24-31eba49e4a41-10.3.121.96-diag.txt.gz
        7.51 MB
        Andrei Baranouski
      5. 3a0ad78b-cf67-4e70-8d24-31eba49e4a41-10.3.121.97-diag.txt.gz
        7.69 MB
        Andrei Baranouski
      6. 3a0ad78b-cf67-4e70-8d24-31eba49e4a41-10.3.121.98-diag.txt.gz
        8.21 MB
        Andrei Baranouski
      7. logs_94_1.tar.gz
        15.14 MB
        Andrei Baranouski
      8. logs_94.tar.gz
        14.93 MB
        Andrei Baranouski
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

          • Assignee:
            jin Jin Lim (Inactive)
            Reporter:
            andreibaranouski Andrei Baranouski
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes