Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7051

[system test] compaction stopped right after it started in idle cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Incomplete
    • Major
    • 2.0
    • 2.0
    • ns_server
    • Security Level: Public
    • centos 6.2 64bit build 2.0.0-1908

    Description

      Cluster information:

      • 8 centos 6.2 64bit server with 4 cores CPU
      • Each server has 32 GB RAM and 400 GB SSD disk.
      • SSD disk format ext4 on /data
      • Each server has its own SSD drive, no disk sharing with other server.
      • Create cluster with 6 nodes installed couchbase server 2.0.0-1908
      • Cluster has 2 buckets, default (12GB with 2 replica) and saslbucket (12GB with 1 replica).
      • Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)

      10.6.2.37
      10.6.2.38
      10.6.2.39
      10.6.2.40
      10.6.2.42
      10.6.2.43

      • Load 15 million items to default bucket and 20 million to saslbucket. Each key has size from 512 bytes to 1024 bytes
      • Queries all 4 views from 2 docs
      • When finish loading items to both bucket, stop all load and queries and let indexing finish initial index
      • Monitor view file size of all nodes, seeing node 39 having largest file size.

      10.6.2.43
      394G 37G 358G 10% /data
      10.6.2.42
      394G 38G 357G 10% /data
      10.6.2.37
      394G 33G 341G 9% /data
      10.6.2.39
      394G 40G 335G 11% /data
      10.6.2.40
      394G 32G 343G 9% /data
      10.6.2.38
      394G 35G 339G 10% /data

      10.6.2.43
      394G 19G 375G 5% /data
      10.6.2.37
      394G 41G 334G 11% /data
      10.6.2.42
      394G 35G 359G 9% /data
      10.6.2.38
      394G 44G 331G 12% /data
      10.6.2.40
      394G 32G 343G 9% /data
      10.6.2.39
      394G 51G 324G 14% /data

      Look at couchdb log on node 39, seeing compaction of index file of both default bucket and saslbucket stopped right after it started.

      [root@localhost logs]# grep 'compaction starting' couchdb.*
      couchdb.1:[couchdb:info,2012-10-29T11:31:57.751,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T11:32:53.905,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T11:33:46.828,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T11:34:56.182,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T11:36:29.144,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T11:37:18.762,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T11:39:22.252,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T11:42:30.332,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T11:44:18.613,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T11:46:57.251,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T11:51:22.159,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T11:55:17.560,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T11:57:47.033,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T12:01:47.524,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T12:09:50.784,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
      couchdb.1:[couchdb:info,2012-10-29T12:19:47.608,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T12:26:05.454,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T12:33:42.037,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T12:43:51.724,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T12:53:39.184,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T12:59:59.687,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T13:09:49.688,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T13:18:55.085,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T13:29:16.317,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T13:38:39.154,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T13:47:34.038,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T13:58:12.741,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T14:10:21.987,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T14:12:20.334,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T14:23:39.115,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T14:38:52.067,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T14:50:28.477,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T15:07:49.139,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T15:28:53.265,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T15:43:46.510,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T15:47:13.724,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
      couchdb.2:[couchdb:info,2012-10-29T15:50:33.835,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
      [root@localhost logs]# grep 'stopped normally' couchdb.*
      couchdb.1:[couchdb:info,2012-10-29T11:30:11.774,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.2117.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:30:11.791,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.2133.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:30:31.536,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.3337.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:30:31.542,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.3346.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:31:57.756,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.7749.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:32:53.906,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.8144.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:33:46.830,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.8536.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:34:56.184,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.8986.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:36:29.149,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.9615.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:37:18.765,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.9961.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:39:22.254,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.14579.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:42:30.333,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.15815.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:44:18.616,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.19027.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:46:57.252,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.23263.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:51:22.160,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.24995.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:55:17.571,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.26550.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T11:57:47.037,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.27545.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T12:01:47.525,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.29098.1> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T12:09:50.832,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.14717.2> stopped normally
      couchdb.1:[couchdb:info,2012-10-29T12:19:47.609,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.16547.5> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T12:26:05.466,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.13632.7> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T12:33:42.038,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.20141.8> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T12:43:51.732,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.27366.8> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T12:53:39.186,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.31725.8> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T12:59:59.688,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.3375.9> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T13:09:49.689,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.10973.9> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T13:18:55.088,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.14519.9> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T13:29:16.321,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.18536.9> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T13:38:39.156,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.22228.9> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T13:47:34.040,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.25716.9> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T13:58:12.743,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.29919.9> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T14:10:21.989,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.2062.16> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T14:12:20.336,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.2827.16> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T14:23:39.121,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.7244.16> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T14:38:52.068,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.13698.16> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T14:50:28.479,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.19645.16> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T15:07:49.140,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.26432.16> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T15:28:53.268,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.1885.17> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T15:43:46.512,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.7724.17> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T15:47:13.726,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.9065.17> stopped normally
      couchdb.2:[couchdb:info,2012-10-29T15:50:33.840,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.10366.17> stopped normally

      Link to manifest file of build 2.0.0-1908 http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1908-rel.rpm.manifest.xml

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty