Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6754

[system test] database compaction did not run in some nodes

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.0
    • 2.0
    • ns_server
    • Security Level: Public
    • centos 6.2 64bit build 2.0.0-1746

    Description

      Cluster information:

      • 8 centos 6.2 64bit server with 4 cores CPU
      • Each server has 32 GB RAM and 400 GB SSD disk.
      • 24.8 GB RAM for couchbase server at each node
      • SSD disk format ext4 on /data
      • Each server has its own drive, no disk sharing with other server.
      • Cluster has 2 buckets, default (12GB) and saslbucket (12GB) and setup cluster with consistent enable.
      • Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
      • Create cluster with 6 nodes installed couchbase server 2.0.0-1746

      10.6.2.37
      10.6.2.38
      10.6.2.39
      10.6.2.40
      10.6.2.42
      10.6.2.43

      • Load 28 million items to both bucket. Each key has size from 512 bytes to 1500 bytes
      • Mutate those keys with new value from 1024 to 1500 bytes
      • Add 2 nodes 10.6.2.44, 10.6.2.45 and remove 2 node 10.6.2.40, 10.6.2.43

      Monitor database size on all nodes, see compaction does not run on node 39 and 42

      rw-r-r-. 1 couchbase couchbase 33M Sep 26 17:55 980.couch.3
      rw-rr-. 1 couchbase couchbase 33M Sep 26 17:55 981.couch.3
      rw-rr-. 1 couchbase couchbase 33M Sep 26 17:55 982.couch.3
      rw-rr-. 1 couchbase couchbase 33M Sep 26 17:55 983.couch.3
      rw-rr-. 1 couchbase couchbase 33M Sep 26 17:55 984.couch.3
      rw-rr-. 1 couchbase couchbase 33M Sep 26 17:55 985.couch.3
      rw-rr-. 1 couchbase couchbase 32M Sep 26 17:56 986.couch.3
      rw-rr-. 1 couchbase couchbase 33M Sep 26 17:56 987.couch.3
      rw-rr-. 1 couchbase couchbase 33M Sep 26 17:55 988.couch.3
      rw-rr-. 1 couchbase couchbase 33M Sep 26 17:56 989.couch.3
      rw-rr-. 1 couchbase couchbase 33M Sep 26 17:55 98.couch.3
      rw-rr-. 1 couchbase couchbase 33M Sep 26 17:56 99.couch.3
      rw-rr-. 1 couchbase couchbase 421M Sep 26 03:00 access.log
      rw-rr-. 1 couchbase couchbase 4.1K Sep 26 15:15 master.couch.3
      rw-rr-. 1 couchbase couchbase 23K Sep 26 17:55 stats.json
      rw-rr-. 1 couchbase couchbase 23K Sep 26 17:54 stats.json.old

      10.6.2.39
      total 61G
      rw-rr-. 1 couchbase couchbase 119M Sep 26 17:55 0.couch.1
      rw-rr-. 1 couchbase couchbase 119M Sep 26 17:56 103.couch.1
      rw-rr-. 1 couchbase couchbase 119M Sep 26 17:55 104.couch.1
      rw-rr-. 1 couchbase couchbase 122M Sep 26 17:56 105.couch.1
      rw-rr-. 1 couchbase couchbase 119M Sep 26 17:55 106.couch.1
      rw-rr-. 1 couchbase couchbase 122M Sep 26 17:55 107.couch.1
      rw-rr-. 1 couchbase couchbase 121M Sep 26 17:55 108.couch.1
      rw-rr-. 1 couchbase couchbase 117M Sep 26 17:55 109.couch.1
      rw-rr-. 1 couchbase couchbase 123M Sep 26 17:55 10.couch.1
      rw-rr-. 1 couchbase couchbase 119M Sep 26 17:56 110.couch.1
      rw-rr-. 1 couchbase couchbase 121M Sep 26 17:55 111.couch.1
      rw-rr-. 1 couchbase couchbase 117M Sep 26 17:55 112.couch.1
      rw-rr-. 1 couchbase couchbase 116M Sep 26 17:55 113.couch.1
      rw-rr-. 1 couchbase couchbase 123M Sep 26 17:55 114.couch.1
      rw-rr-. 1 couchbase couchbase 122M Sep 26 17:55 115.couch.1
      rw-rr-. 1 couchbase couchbase 118M Sep 26 17:56 116.couch.1
      rw-rr-. 1 couchbase couchbase 119M Sep 26 17:55 117.couch.1
      rw-rr-. 1 couchbase couchbase 121M Sep 26 17:55 118.couch.1
      rw-rr-. 1 couchbase couchbase 120M Sep 26 17:55 119.couch.1
      rw-rr-. 1 couchbase couchbase 119M Sep 26 17:55 11.

      10.6.2.42
      total 30G
      rw-rr-. 1 couchbase couchbase 138M Sep 26 17:55 1000.couch.2
      rw-rr-. 1 couchbase couchbase 134M Sep 26 17:56 1001.couch.2
      rw-rr-. 1 couchbase couchbase 137M Sep 26 17:56 1002.couch.2
      rw-rr-. 1 couchbase couchbase 135M Sep 26 17:55 1003.couch.2
      rw-rr-. 1 couchbase couchbase 135M Sep 26 17:55 1004.couch.2
      rw-rr-. 1 couchbase couchbase 133M Sep 26 17:55 1005.couch.2
      rw-rr-. 1 couchbase couchbase 137M Sep 26 17:55 1006.couch.2
      rw-rr-. 1 couchbase couchbase 137M Sep 26 17:56 1007.couch.2
      rw-rr-. 1 couchbase couchbase 133M Sep 26 17:56 1008.couch.2
      rw-rr-. 1 couchbase couchbase 135M Sep 26 17:56 1009.couch.2
      rw-rr-. 1 couchbase couchbase 133M Sep 26 17:56 1010.couch.2
      rw-rr-. 1 couchbase couchbase 140M Sep 26 17:55 1011.couch.2
      rw-rr-. 1 couchbase couchbase 137M Sep 26 17:56 1012.couch.2
      rw-rr-. 1 couchbase couchbase 137M Sep 26 17:56 1013.couch.2
      rw-rr-. 1 couchbase couchbase 136M Sep 26 17:55 1014.couch.2
      rw-rr-. 1 couchbase couchbase 134M Sep 26 17:55 1015.couch.2

      Database on node 39 and 42 keep going up

      10.6.2.40
      394G 111G 263G 30% /data
      10.6.2.37
      394G 114G 261G 31% /data
      10.6.2.45
      394G 8.1G 386G 3% /data
      10.6.2.39
      394G 308G 67G 83% /data
      10.6.2.43
      394G 128G 266G 33% /data
      10.6.2.42
      394G 276G 119G 70% /data
      10.6.2.44
      394G 12G 383G 3% /data
      10.6.2.38
      394G 183G 192G 49% /data

      Manually trigger compaction by change setting to 20% (from default value 30%), see compaction start running on node 42, not node 39.
      This may lead to node 39 crashed due to run out of space

      10.6.2.42
      total 16G
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1000.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1001.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1002.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:04 1003.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:04 1004.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1005.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1006.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1007.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:04 1008.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1009.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1010.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1011.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1012.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1013.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1014.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1015.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1016.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1017.couch.3
      rw-rr-. 1 couchbase couchbase 30M Sep 26 18:05 1018.couch.3
      rw-rr-. 1 couchbase couchbase 135M Sep 26 18:05 1019.couch.2
      rw-rr-. 1 couchbase couchbase 24M Sep 26 18:05 1019.couch.2.compact
      rw-rr-. 1 couchbase couchbase 136M Sep 26 18:05 1020.couch.2
      rw-rr-. 1 couchbase couchbase 141M Sep 26 18:05 1021.couch.2
      rw-rr-. 1 couchbase couchbase 139M Sep 26 18:05 1022.couch.2
      rw-rr-. 1 couchbase couchbase 137M Sep 26 18:04 1023.couch.2

      Thuans-MacBook-Pro:testrunner thuan$ python scripts/ssh.py -i ../ini/10-c-long.ini "df -kh | grep data"
      10.6.2.40
      394G 116G 258G 32% /data

      10.6.2.43
      394G 120G 274G 31% /data

      10.6.2.39
      394G 314G 61G 84% /data

      10.6.2.37
      394G 119G 256G 32% /data

      10.6.2.45
      394G 9.1G 385G 3% /data

      10.6.2.44
      394G 7.0G 387G 2% /data

      10.6.2.42
      394G 253G 142G 65% /data

      10.6.2.38
      394G 186G 189G 50% /data

      Thuans-MacBook-Pro:testrunner thuan$ python scripts/ssh.py -i ../ini/10-c-long.ini "df -kh | grep data"
      10.6.2.45
      394G 8.5G 386G 3% /data

      10.6.2.40
      394G 118G 257G 32% /data

      10.6.2.43
      394G 118G 277G 30% /data

      10.6.2.44
      394G 7.7G 387G 2% /data

      10.6.2.39
      394G 315G 60G 85% /data

      10.6.2.37
      394G 120G 254G 33% /data

      10.6.2.42
      394G 254G 140G 65% /data

      10.6.2.38
      394G 185G 190G 50% /data

      Link to collect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201209/8nodes-col-info-1746-database-compaction-didnot-run-in-some-nodes-20120926-180930.tgz

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            thuan Thuan Nguyen
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty