Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.5.0
-
Untriaged
-
Unknown
Description
Build
4.5.0-2428
Testcase
test_4 in dgm suite
./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,GROUP=DGM -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,cluster=D+F,F,F,dgm_run=1,active_resident_ratio=10,standard_buckets=1,sasl_buckets=1,GROUP=DGM
1. cluster contains 1 data+fts node, 2 more fts nodes.
2. 3 buckets, loaded to active_resident_ratio 10%
3. On creation of second full text index, .48 becomes unresponsive with pools/default returning "unexpected server error".
Node configuration:
8GB RAM, 4 cores, 200GB SSDs
|
[2016-05-04 18:33:31,924] - [fts_base:609] INFO - Creating fulltext-index sasl_bucket_1_index_1 on 172.23.106.47
|
[2016-05-04 18:33:31,924] - [rest_client:2158] INFO - {"params": {}, "name": "sasl_bucket_1_index_1", "planParams": {"hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32}, "sourceName": "sasl_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {"authUser": "sasl_bucket_1", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "password", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0}, "uuid": ""}
|
[2016-05-04 18:33:32,346] - [rest_client:2167] INFO - Index sasl_bucket_1_index_1 created
|
[2016-05-04 18:33:32,347] - [fts_base:2472] INFO - Validating index distribution for sasl_bucket_1_index_1 ...
|
[2016-05-04 18:33:32,660] - [fts_base:2383] INFO - sleep for 5 secs. No pindexes found, waiting for index to get created ...
|
[2016-05-04 18:35:38,385] - [fts_base:2490] INFO - Validated: Number of PIndexes = 32
|
[2016-05-04 18:35:39,244] - [rest_client:775] ERROR - http://172.23.106.48:8091/pools/default error 500 reason: unknown ["Unexpected server error, request logged."]
|
[2016-05-04 18:35:39,245] - [fts_base:2501] INFO - Validated: Every pIndex serves 32 partitions or lesser
|
[2016-05-04 18:35:39,246] - [fts_base:2383] INFO - sleep for 10 secs. pIndexes not distributed across 0 nodes yet ...
|
[2016-05-04 18:35:49,260] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:35:52,266] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:35:55,274] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:35:58,280] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:01,286] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:04,292] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:07,297] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:10,303] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:13,309] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:16,315] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:19,322] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:22,328] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:25,334] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:28,343] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:31,349] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:34,355] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:37,360] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:40,367] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:43,373] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
[2016-05-04 18:36:46,379] - [rest_client:778] ERROR - socket error while connecting to http://172.23.106.48:8091/nodes/self error [Errno 111] Connection refused
|
.48 is unavailable all of a sudden. Attaching cbcollect info from all 3 nodes.