Details
-
Bug
-
Resolution: Fixed
-
Major
-
2.0, 2.0.1, 2.1.0
-
Security Level: Public
-
centos5.6, build 1952 with 16 scheduler
-
Untriaged
Description
Cluster: 6 nodes
10.6.2.37
10.6.2.38
10.6.2.39
10.6.2.40
10.6.2.42
10.6.2.43
Build # 2.0.0-1952 with 16 erlang schedulers
each nodes with 390GB SSD drive, 32GB RAM
2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.
I don't put any limit on query results and those queries are generated without waiting for previous ones finish showing the results:
capiUrl = "http://%s:%s/couchBase/" % (cfg.COUCHBASE_IP, cfg.COUCHBASE_PORT)
url = capiUrl + '%s/design/%s/%s/%s' % (bucket,
design_doc_name, type_,
view_name)
headers =
req = urllib2.Request(url, headers = headers)
Then the UI becomes unresponsive.
Pay attention to the following stats:
1st is the erlang scheduler on one of the nodes during query happens:
(ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
#Fun<erl_eval.6.80247286>
1353032384137
1353032384293
{4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}1353032384425
{3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}1353032384553
{23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}1353032384672
{16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}1353032384795
{6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}1353032384919
{1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}1353032385081
{54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}1353032385221
{15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}1353032385355
{46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}1353032385468
{11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}1353032385610
{7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}1353032385765
{7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}1353032385905
{9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}1353032386068
{48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}1353032386222
{11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}1353032386356
{64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}1353032386560
{0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}1353032386700
{50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}1353032386837
{0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}1353032386984
{2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}1353032387105
{1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}1353032387231
{22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}1353032387337
{17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}1353032387469
{5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}1353032387598
{2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}1353032387746
{2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}1353032387897
{3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}1353032388021
{29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}1353032388146
{15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}1353032388277
{53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}1353032388402
{2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}1353032388594
{17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}1353032388734
{35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}1353032388873
{2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}1353032389008
{112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}1353032389133
{102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}1353032389257
{44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}1353032389379
{4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}1353032389549
{24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}1353032389686
{54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}1353032389804
{79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}1353032389950
{90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}1353032390101
{59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}2nd is the top stats about beam.smp:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp
24G memory usage. And the CPU% is always above 350%