Blocker Description
Execute the following test case:
./testrunner -i centos_x64--rebalance_out.ini -t 2i.indexscans_2i.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index_with_concurrent_mutations,groups=simple,dataset=default,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc-per-day=200,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,sasl_buckets=1,standard_buckets=1,gsi_type=forestdb,services_init=kv:n1ql-kv:index-index-index,nodes_init=4
|
After some time there are timeout errors, There are about 7 queries running in parallel:
2016-04-02 07:55:05 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : scan_consistency=request_plus&statement=SELECT+%2A+FROM+bucket0+WHERE++join_yr+%3E+1999+ORDER+BY+_id+
|
2016-04-02 07:56:43 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://10.6.2.144:8093/query?scan_consistency=request_plus&statement=SELECT+%2A+FROM+bucket0+WHERE++join_yr+%3E+1999+ORDER+BY+_id+ error [Errno 111] Connection refused
|
2016-04-02 07:58:51 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://10.6.2.144:8093/query?scan_consistency=request_plus&statement=SELECT+%2A+FROM+bucket0+WHERE++join_yr+%3E+1999+ORDER+BY+_id+ error [Errno 111] Connection refused
|
2016-04-02 08:01:07 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://10.6.2.144:8093/query?scan_consistency=request_plus&statement=SELECT+%2A+FROM+bucket0+WHERE++join_yr+%3E+1999+ORDER+BY+_id+ error [Errno 111] Connection refused
|
2016-04-02 08:01:10 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://10.6.2.144:8093/query?scan_consistency=request_plus&statement=SELECT+%2A+FROM+bucket0+WHERE++join_yr+%3E+1999+ORDER+BY+_id+ error [Errno 111] Connection refused
|
2016-04-02 08:03:16 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://10.6.2.144:8093/query?scan_consistency=request_plus&statement=SELECT+%2A+FROM+bucket0+WHERE++join_yr+%3E+1999+ORDER+BY+_id+ error [Errno 111] Connection refused
|
2016-04-02 08:03:19 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://10.6.2.144:8093/query?scan_consistency=request_plus&statement=SELECT+%2A+FROM+bucket0+WHERE++join_yr+%3E+1999+ORDER+BY+_id+ error [Errno 111] Connection refused
|
2016-04-02 08:03:22 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://10.6.2.144:8093/query?scan_consistency=request_plus&statement=SELECT+%2A+FROM+bucket0+WHERE++join_yr+%3E+1999+ORDER+BY+_id+ error [Errno 111] Connection refused
|
2016-04-02 08:03:25 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://10.6.2.144:8093/query?scan_consistency=request_plus&statement=SELECT+%2A+FROM+bucket0+WHERE++join_yr+%3E+1999+ORDER+BY+_id+ error [Errno 111] Connection refused
|
UI logs show query service restarting:
Service 'query' exited with status 1. Restarting. Messages: _time=2016-04-02T08:20:39.543-07:00 _level=INFO _msg=Got new configuration for bucket standard_bucket0
|
2016-04-02T08:20:39.726-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification
|
_time=2016-04-02T08:20:39.850-07:00 _level=INFO _msg=Got new configuration for bucket default
|
_time=2016-04-02T08:20:39.883-07:00 _level=INFO _msg=Got new configuration for bucket bucket0
|
[goport] 2016/04/02 08:20:43 /opt/couchbase/bin/cbq-engine terminated: signal: killed ns_log 000 ns_1@10.6.2.144 8:20:43 AM Sat Apr 2, 2016
|
Haven't heard from a higher priority node or a master, so I'm taking over. (repeated 1 times) mb_master 000 ns_1@10.6.2.145 8:20:37 AM Sat Apr 2, 2016
|
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master 000 ns_1@10.6.2.145 8:19:53 AM Sat Apr 2, 2016
|
Service 'query' exited with status 1. Restarting. Messages: 2016-04-02T08:08:03.08-07:00 [Info] index 10783981039385679021 has 1 replicas
|
2016-04-02T08:08:04.121-07:00 [Info] index 12435147937827683140 has 1 replicas
|
2016-04-02T08:08:05.134-07:00 [Info] index 17677069678706168633 has 1 replicas
|
2016-04-02T08:08:06.295-07:00 [Info] client load stats {"4211994321797953933": 1.1796064494e+10}
|
[goport] 2016/04/02 08:08:23 /opt/couchbase/bin/cbq-engine terminated: signal: killed ns_log 000 ns_1@10.6.2.144 8:08:24 AM Sat Apr 2, 2016
|
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master 000 ns_1@10.6.2.145 8:08:19 AM Sat Apr 2, 2016
|
Service 'query' exited with status 1. Restarting. Messages: 2016-04-02T08:05:31.095-07:00 [Info] index 281393133082369571 has 1 replicas
|
2016-04-02T08:05:31.095-07:00 [Info] index 10733920939502971404 has 1 replicas
|
2016-04-02T08:05:31.095-07:00 [Info] index 1696672998557284032 has 1 replicas
|
2016-04-02T08:05:31.183-07:00 [Info] client load stats {"4211994321797953933": 1.2058964145e+10}
|
[goport] 2016/04/02 08:05:40 /opt/couchbase/bin/cbq-engine terminated: signal: killed ns_log 000 ns_1@10.6.2.144 8:05:40 AM Sat Apr 2, 2016
|
Service 'query' exited with status 1. Restarting. Messages: 2016-04-02T08:02:13.981-07:00 [Info] index 1696672998557284032 has 1 replicas
|
2016-04-02T08:02:13.981-07:00 [Info] index 10733920939502971404 has 1 replicas
|
2016-04-02T08:02:13.981-07:00 [Info] client load stats {"4211994321797953933": 1.1638265284e+10}
|
2016-04-02T08:02:30.883-07:00 [Info] GSIC[default/bucket0-1459609273854519501] Scan(3a1c9c43-9654-46f0-bba2-4c034f3b8736) removing backfill file /tmp/scan-backfill10837368235879 ...
|
[goport] 2016/04/02 08:03:16 /opt/couchbase/bin/cbq-engine terminated: signal: killed ns_log 000 ns_1@10.6.2.144 8:03:17 AM Sat Apr 2, 2016
|
Service 'query' exited with status 1. Restarting. Messages: 2016-04-02T08:01:01.584-07:00 [Info] index 10783981039385679021 has 1 replicas
|
2016-04-02T08:01:01.584-07:00 [Info] index 281393133082369571 has 1 replicas
|
2016-04-02T08:01:01.584-07:00 [Info] index 10733920939502971404 has 1 replicas
|
2016-04-02T08:01:01.584-07:00 [Info] index 1696672998557284032 has 1 replicas
|
[goport] 2016/04/02 08:01:08 /opt/couchbase/bin/cbq-engine terminated: signal: killed ns_log 000 ns_1@10.6.2.144 8:01:08 AM Sat Apr 2, 2016
|
Service 'query' exited with status 1. Restarting. Messages: 2016-04-02T07:57:46.669-07:00 [Info] index 281393133082369571 has 1 replicas
|
2016-04-02T07:57:46.669-07:00 [Info] client load stats {"4211994321797953933": 1.1574509267e+10}
|
2016-04-02T07:57:46.671-07:00 [Info] GSIC[default/bucket0-1459609006583887707] logstats "bucket0" {"gsi_scan_count":1,"gsi_scan_duration":11574682880,"gsi_throttle_duration":2490132730,"gsi_prime_duration":23522379,"gsi_blocked_duration":56313491376,"gsi_totalbackfills":0}
|
2016-04-02T07:58:13.56-07:00 [Info] GSIC[default/bucket0-1459609006583887707] Scan(cfeae50c-0e2e-440d-ad70-8262773db5b3) removing backfill file /tmp/scan-backfill10616439678670 ...
|
[goport] 2016/04/02 07:58:51 /opt/couchbase/bin/cbq-engine terminated: signal: killed ns_log 000 ns_1@10.6.2.144 7:58:51 AM Sat Apr 2, 2016
|
Service 'query' exited with status 1. Restarting. Messages: 2016-04-02T07:55:41.729-07:00 [Info] index 10733920939502971404 has 1 replicas
|
2016-04-02T07:55:41.729-07:00 [Info] index 1696672998557284032 has 1 replicas
|
2016-04-02T07:55:41.729-07:00 [Info] index 281393133082369571 has 1 replicas
|
2016-04-02T07:55:41.729-07:00 [Info] client load stats {"10733920939502971404": 4.303343123e+09,"4211994321797953933": 1.1972452417e+10}
|
[goport] 2016/04/02 07:56:43 /opt/couchbase/bin/cbq-engine terminated: signal: killed ns_log 000 ns_1@10.6.2.144 7:56:43 AM Sat Apr 2, 2016
|
Service 'query' exited with status 1. Restarting. Messages: 2016-04-02T07:52:50.451-07:00 [Info] index 1696672998557284032 has 1 replicas
|
2016-04-02T07:52:50.451-07:00 [Info] index 10733920939502971404 has 1 replicas
|
2016-04-02T07:52:50.452-07:00 [Info] client load stats {"10733920939502971404": 6.612007466e+09}
|
2016-04-02T07:52:50.453-07:00 [Info] GSIC[default/bucket0-1459608710274690370] logstats "bucket0" {"gsi_scan_count":2,"gsi_scan_duration":13224378105,"gsi_throttle_duration":2697959518,"gsi_prime_duration":4169442899,"gsi_blocked_duration":94192036648,"gsi_totalbackfills":2}
|
[goport] 2016/04/02 07:53:38 /opt/couchbase/bin/cbq-engine terminated: signal: killed ns_log 000 ns_1@10.6.2.144 7:53:38 AM Sat Apr 2, 2016
|
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master 000 ns_1@10.6.2.145 7:53:11 AM Sat Apr 2, 2016
|
Service 'query' exited with status 1. Restarting. Messages: 2016-04-02T07:51:03.497-07:00 [Info] index 281393133082369571 has 1 replicas
|
2016-04-02T07:51:03.497-07:00 [Info] client load stats {"10733920939502971404": 2.5671467914e+10}
|
2016-04-02T07:51:11.915-07:00 [Info] GSIC[default/bucket0-1459606083390340264] Scan(94b6128d-3dc1-4096-a728-59da2d9867dc) removing backfill file /tmp/scan-backfill7231175137695 ...
|
2016-04-02T07:51:13.291-07:00 [Info] GSIC[default/bucket0-1459606083390340264] Scan(94b6128d-3dc1-4096-a728-59da2d9867dc) removing backfill file /tmp/scan-backfill7231388992626 ...
|
[goport] 2016/04/02 07:51:46 /opt/couchbase/bin/cbq-engine terminated: signal: killed
|