Details
-
Bug
-
Resolution: Incomplete
-
Major
-
2.0
-
Security Level: Public
-
centos 6.2 64bit build 2.0.0-1908
Description
Cluster information:
- 8 centos 6.2 64bit server with 4 cores CPU
- Each server has 32 GB RAM and 400 GB SSD disk.
- SSD disk format ext4 on /data
- Each server has its own SSD drive, no disk sharing with other server.
- Create cluster with 6 nodes installed couchbase server 2.0.0-1908
- Cluster has 2 buckets, default (12GB with 2 replica) and saslbucket (12GB with 1 replica).
- Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
10.6.2.37
10.6.2.38
10.6.2.39
10.6.2.40
10.6.2.42
10.6.2.43
- Load 15 million items to default bucket and 20 million to saslbucket. Each key has size from 512 bytes to 1024 bytes
- Queries all 4 views from 2 docs
- When finish loading items to both bucket, stop all load and queries and let indexing finish initial index
- Monitor view file size of all nodes, seeing node 39 having largest file size.
10.6.2.43
394G 37G 358G 10% /data
10.6.2.42
394G 38G 357G 10% /data
10.6.2.37
394G 33G 341G 9% /data
10.6.2.39
394G 40G 335G 11% /data
10.6.2.40
394G 32G 343G 9% /data
10.6.2.38
394G 35G 339G 10% /data
10.6.2.43
394G 19G 375G 5% /data
10.6.2.37
394G 41G 334G 11% /data
10.6.2.42
394G 35G 359G 9% /data
10.6.2.38
394G 44G 331G 12% /data
10.6.2.40
394G 32G 343G 9% /data
10.6.2.39
394G 51G 324G 14% /data
Look at couchdb log on node 39, seeing compaction of index file of both default bucket and saslbucket stopped right after it started.
[root@localhost logs]# grep 'compaction starting' couchdb.*
couchdb.1:[couchdb:info,2012-10-29T11:31:57.751,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T11:32:53.905,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T11:33:46.828,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T11:34:56.182,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T11:36:29.144,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T11:37:18.762,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T11:39:22.252,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T11:42:30.332,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T11:44:18.613,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T11:46:57.251,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T11:51:22.159,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T11:55:17.560,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T11:57:47.033,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T12:01:47.524,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T12:09:50.784,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
couchdb.1:[couchdb:info,2012-10-29T12:19:47.608,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T12:26:05.454,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T12:33:42.037,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T12:43:51.724,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T12:53:39.184,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T12:59:59.687,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T13:09:49.688,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T13:18:55.085,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T13:29:16.317,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T13:38:39.154,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T13:47:34.038,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T13:58:12.741,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T14:10:21.987,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T14:12:20.334,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T14:23:39.115,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T14:38:52.067,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T14:50:28.477,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T15:07:49.139,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T15:28:53.265,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T15:43:46.510,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T15:47:13.724,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, compaction starting
couchdb.2:[couchdb:info,2012-10-29T15:50:33.835,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, compaction starting
[root@localhost logs]# grep 'stopped normally' couchdb.*
couchdb.1:[couchdb:info,2012-10-29T11:30:11.774,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.2117.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:30:11.791,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.2133.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:30:31.536,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.3337.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:30:31.542,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.3346.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:31:57.756,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.7749.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:32:53.906,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.8144.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:33:46.830,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.8536.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:34:56.184,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.8986.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:36:29.149,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.9615.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:37:18.765,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.9961.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:39:22.254,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.14579.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:42:30.333,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.15815.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:44:18.616,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.19027.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:46:57.252,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.23263.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:51:22.160,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.24995.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:55:17.571,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.26550.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T11:57:47.037,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.27545.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T12:01:47.525,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.29098.1> stopped normally
couchdb.1:[couchdb:info,2012-10-29T12:09:50.832,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.14717.2> stopped normally
couchdb.1:[couchdb:info,2012-10-29T12:19:47.609,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.16547.5> stopped normally
couchdb.2:[couchdb:info,2012-10-29T12:26:05.466,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.13632.7> stopped normally
couchdb.2:[couchdb:info,2012-10-29T12:33:42.038,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.20141.8> stopped normally
couchdb.2:[couchdb:info,2012-10-29T12:43:51.732,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.27366.8> stopped normally
couchdb.2:[couchdb:info,2012-10-29T12:53:39.186,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.31725.8> stopped normally
couchdb.2:[couchdb:info,2012-10-29T12:59:59.688,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.3375.9> stopped normally
couchdb.2:[couchdb:info,2012-10-29T13:09:49.689,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.10973.9> stopped normally
couchdb.2:[couchdb:info,2012-10-29T13:18:55.088,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.14519.9> stopped normally
couchdb.2:[couchdb:info,2012-10-29T13:29:16.321,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.18536.9> stopped normally
couchdb.2:[couchdb:info,2012-10-29T13:38:39.156,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.22228.9> stopped normally
couchdb.2:[couchdb:info,2012-10-29T13:47:34.040,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.25716.9> stopped normally
couchdb.2:[couchdb:info,2012-10-29T13:58:12.743,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.29919.9> stopped normally
couchdb.2:[couchdb:info,2012-10-29T14:10:21.989,ns_1@10.6.2.39:<0.3336.1>:couch_log:info:39]Set view `default`, main group `_design/d1`, linked PID <0.2062.16> stopped normally
couchdb.2:[couchdb:info,2012-10-29T14:12:20.336,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.2827.16> stopped normally
couchdb.2:[couchdb:info,2012-10-29T14:23:39.121,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.7244.16> stopped normally
couchdb.2:[couchdb:info,2012-10-29T14:38:52.068,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.13698.16> stopped normally
couchdb.2:[couchdb:info,2012-10-29T14:50:28.479,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.19645.16> stopped normally
couchdb.2:[couchdb:info,2012-10-29T15:07:49.140,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.26432.16> stopped normally
couchdb.2:[couchdb:info,2012-10-29T15:28:53.268,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.1885.17> stopped normally
couchdb.2:[couchdb:info,2012-10-29T15:43:46.512,ns_1@10.6.2.39:<0.2116.1>:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, linked PID <0.7724.17> stopped normally
couchdb.2:[couchdb:info,2012-10-29T15:47:13.726,ns_1@10.6.2.39:<0.2125.1>:couch_log:info:39]Set view `saslbucket`, replica group `_design/d11`, linked PID <0.9065.17> stopped normally
couchdb.2:[couchdb:info,2012-10-29T15:50:33.840,ns_1@10.6.2.39:<0.3345.1>:couch_log:info:39]Set view `default`, replica group `_design/d1`, linked PID <0.10366.17> stopped normally
Link to manifest file of build 2.0.0-1908 http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1908-rel.rpm.manifest.xml