Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7199

Couchbase server can't handle hundreds of view queries with unlimited number of results at the same time

    Details

    • Triage:
      Untriaged

      Description

      Cluster: 6 nodes
      10.6.2.37
      10.6.2.38
      10.6.2.39
      10.6.2.40
      10.6.2.42
      10.6.2.43

      Build # 2.0.0-1952 with 16 erlang schedulers
      each nodes with 390GB SSD drive, 32GB RAM

      2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.

      I don't put any limit on query results and those queries are generated without waiting for previous ones finish showing the results:
      capiUrl = "http://%s:%s/couchBase/" % (cfg.COUCHBASE_IP, cfg.COUCHBASE_PORT)
      url = capiUrl + '%s/design/%s/%s/%s' % (bucket,
      design_doc_name, type_,
      view_name)
      headers =

      {'Content-Type': 'application/json', 'Authorization': 'Basic %s' % authorization, 'Accept': '*/*'}

      req = urllib2.Request(url, headers = headers)

      Then the UI becomes unresponsive.
      Pay attention to the following stats:

      1st is the erlang scheduler on one of the nodes during query happens:

      (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
      #Fun<erl_eval.6.80247286>
      1353032384137

      {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}

      1353032384293

      {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}

      1353032384425

      {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}

      1353032384553

      {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}

      1353032384672

      {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}

      1353032384795

      {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}

      1353032384919

      {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}

      1353032385081

      {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}

      1353032385221

      {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}

      1353032385355

      {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}

      1353032385468

      {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}

      1353032385610

      {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}

      1353032385765

      {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}

      1353032385905

      {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}

      1353032386068

      {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}

      1353032386222

      {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}

      1353032386356

      {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}

      1353032386560

      {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}

      1353032386700

      {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}

      1353032386837

      {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}

      1353032386984

      {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}

      1353032387105

      {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}

      1353032387231

      {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}

      1353032387337

      {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}

      1353032387469

      {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}

      1353032387598

      {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}

      1353032387746

      {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}

      1353032387897

      {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}

      1353032388021

      {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}

      1353032388146

      {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}

      1353032388277

      {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}

      1353032388402

      {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}

      1353032388594

      {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}

      1353032388734

      {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}

      1353032388873

      {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}

      1353032389008

      {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}

      1353032389133

      {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}

      1353032389257

      {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}

      1353032389379

      {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}

      1353032389549

      {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}

      1353032389686

      {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}

      1353032389804

      {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}

      1353032389950

      {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}

      1353032390101

      {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}

      2nd is the top stats about beam.smp:
      PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
      676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp

      24G memory usage. And the CPU% is always above 350%

      1. erl_crash.dump.tgz
        157 kB
        FilipeManana
      2. logs.tgz
        3.23 MB
        FilipeManana
      3. queries2.png
        75 kB
        Tommie McAfee
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Chisheng Chisheng Hong (Inactive) created issue -
        Chisheng Chisheng Hong (Inactive) made changes -
        Field Original Value New Value
        Description Cluster: 6 nodes
        10.6.2.37
        10.6.2.38
        10.6.2.39
        10.6.2.40
        10.6.2.42
        10.6.2.43

        Build # 2.0.0-1952 with 16 erlang schedulers
        each nodes with 390GB SSD drive, 32GB RAM

        2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.

        Then the UI becomes unresponsive.
        Pay attention to the following stats:

        1st is the erlang scheduler on one of the nodes during query happens:

        (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
        #Fun<erl_eval.6.80247286>
        1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}
        1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}
        1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}
        1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}
        1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}
        1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}
        1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}
        1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}
        1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}
        1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}
        1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}
        1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}
        1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}
        1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}
        1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}
        1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}
        1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}
        1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}
        1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}
        1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}
        1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}
        1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}
        1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}
        1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}
        1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}
        1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}
        1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}
        1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}
        1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}
        1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}
        1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}
        1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}
        1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}
        1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}
        1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}
        1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}
        1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}
        1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}
        1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}
        1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}
        1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}
        1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}

        2nd is the top stats about beam.smp:
          PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
          676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp

        24G memory usage. And the CPU% is always above 350%


        erl_crash.dmp location: root:couchbase@10.3.2.37:/opt/couchbase/bin/erl_crash.dump.11-15-2012-17\:14\:38.651



         
        Cluster: 6 nodes
        10.6.2.37
        10.6.2.38
        10.6.2.39
        10.6.2.40
        10.6.2.42
        10.6.2.43

        Build # 2.0.0-1952 with 16 erlang schedulers
        each nodes with 390GB SSD drive, 32GB RAM

        2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.

        Then the UI becomes unresponsive.
        Pay attention to the following stats:

        1st is the erlang scheduler on one of the nodes during query happens:

        (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
        #Fun<erl_eval.6.80247286>
        1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}
        1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}
        1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}
        1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}
        1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}
        1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}
        1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}
        1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}
        1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}
        1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}
        1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}
        1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}
        1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}
        1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}
        1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}
        1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}
        1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}
        1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}
        1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}
        1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}
        1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}
        1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}
        1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}
        1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}
        1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}
        1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}
        1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}
        1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}
        1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}
        1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}
        1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}
        1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}
        1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}
        1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}
        1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}
        1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}
        1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}
        1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}
        1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}
        1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}
        1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}
        1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}

        2nd is the top stats about beam.smp:
          PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
          676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp

        24G memory usage. And the CPU% is always above 350%


        erl_crash.dmp location: root:couchbase@10.6.2.37:/opt/couchbase/bin/erl_crash.dump.11-15-2012-17\:14\:38.651



         
        Chisheng Chisheng Hong (Inactive) made changes -
        Description Cluster: 6 nodes
        10.6.2.37
        10.6.2.38
        10.6.2.39
        10.6.2.40
        10.6.2.42
        10.6.2.43

        Build # 2.0.0-1952 with 16 erlang schedulers
        each nodes with 390GB SSD drive, 32GB RAM

        2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.

        Then the UI becomes unresponsive.
        Pay attention to the following stats:

        1st is the erlang scheduler on one of the nodes during query happens:

        (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
        #Fun<erl_eval.6.80247286>
        1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}
        1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}
        1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}
        1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}
        1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}
        1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}
        1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}
        1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}
        1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}
        1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}
        1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}
        1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}
        1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}
        1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}
        1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}
        1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}
        1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}
        1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}
        1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}
        1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}
        1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}
        1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}
        1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}
        1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}
        1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}
        1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}
        1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}
        1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}
        1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}
        1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}
        1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}
        1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}
        1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}
        1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}
        1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}
        1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}
        1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}
        1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}
        1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}
        1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}
        1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}
        1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}

        2nd is the top stats about beam.smp:
          PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
          676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp

        24G memory usage. And the CPU% is always above 350%


        erl_crash.dmp location: root:couchbase@10.6.2.37:/opt/couchbase/bin/erl_crash.dump.11-15-2012-17\:14\:38.651



         
        Cluster: 6 nodes
        10.6.2.37
        10.6.2.38
        10.6.2.39
        10.6.2.40
        10.6.2.42
        10.6.2.43

        Build # 2.0.0-1952 with 16 erlang schedulers
        each nodes with 390GB SSD drive, 32GB RAM

        2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.

        Then the UI becomes unresponsive.
        Pay attention to the following stats:

        1st is the erlang scheduler on one of the nodes during query happens:

        (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
        #Fun<erl_eval.6.80247286>
        1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}
        1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}
        1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}
        1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}
        1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}
        1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}
        1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}
        1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}
        1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}
        1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}
        1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}
        1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}
        1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}
        1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}
        1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}
        1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}
        1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}
        1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}
        1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}
        1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}
        1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}
        1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}
        1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}
        1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}
        1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}
        1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}
        1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}
        1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}
        1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}
        1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}
        1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}
        1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}
        1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}
        1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}
        1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}
        1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}
        1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}
        1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}
        1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}
        1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}
        1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}
        1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}

        2nd is the top stats about beam.smp:
          PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
          676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp

        24G memory usage. And the CPU% is always above 350%


        erl_crash.dmp location: root:couchbase@10.6.2.37:/opt/couchbase/var/lib/couchbase/erl_crash.dump.11-15-2012-17\:14\:38.651



         
        tommie Tommie McAfee made changes -
        Attachment queries2.png [ 15835 ]
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] Aliaksey Artamonau [ aliaksey artamonau ]
        Aliaksey Artamonau Aliaksey Artamonau made changes -
        Assignee Aliaksey Artamonau [ aliaksey artamonau ] Tommie McAfee [ tommie ]
        Chisheng Chisheng Hong (Inactive) made changes -
        Description Cluster: 6 nodes
        10.6.2.37
        10.6.2.38
        10.6.2.39
        10.6.2.40
        10.6.2.42
        10.6.2.43

        Build # 2.0.0-1952 with 16 erlang schedulers
        each nodes with 390GB SSD drive, 32GB RAM

        2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.

        Then the UI becomes unresponsive.
        Pay attention to the following stats:

        1st is the erlang scheduler on one of the nodes during query happens:

        (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
        #Fun<erl_eval.6.80247286>
        1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}
        1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}
        1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}
        1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}
        1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}
        1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}
        1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}
        1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}
        1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}
        1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}
        1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}
        1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}
        1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}
        1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}
        1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}
        1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}
        1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}
        1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}
        1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}
        1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}
        1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}
        1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}
        1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}
        1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}
        1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}
        1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}
        1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}
        1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}
        1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}
        1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}
        1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}
        1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}
        1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}
        1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}
        1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}
        1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}
        1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}
        1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}
        1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}
        1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}
        1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}
        1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}

        2nd is the top stats about beam.smp:
          PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
          676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp

        24G memory usage. And the CPU% is always above 350%


        erl_crash.dmp location: root:couchbase@10.6.2.37:/opt/couchbase/var/lib/couchbase/erl_crash.dump.11-15-2012-17\:14\:38.651



         
        Cluster: 6 nodes
        10.6.2.37
        10.6.2.38
        10.6.2.39
        10.6.2.40
        10.6.2.42
        10.6.2.43

        Build # 2.0.0-1952 with 16 erlang schedulers
        each nodes with 390GB SSD drive, 32GB RAM

        2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.

        Then the UI becomes unresponsive.
        Pay attention to the following stats:

        1st is the erlang scheduler on one of the nodes during query happens:

        (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
        #Fun<erl_eval.6.80247286>
        1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}
        1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}
        1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}
        1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}
        1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}
        1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}
        1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}
        1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}
        1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}
        1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}
        1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}
        1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}
        1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}
        1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}
        1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}
        1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}
        1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}
        1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}
        1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}
        1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}
        1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}
        1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}
        1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}
        1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}
        1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}
        1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}
        1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}
        1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}
        1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}
        1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}
        1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}
        1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}
        1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}
        1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}
        1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}
        1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}
        1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}
        1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}
        1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}
        1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}
        1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}
        1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}

        2nd is the top stats about beam.smp:
          PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
          676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp

        24G memory usage. And the CPU% is always above 350%
        farshid Farshid Ghods (Inactive) made changes -
        Assignee Tommie McAfee [ tommie ] Chisheng Hong [ chisheng ]
        Chisheng Chisheng Hong (Inactive) made changes -
        Summary Couchbase server can't handle view query with S 16:16 with build 1952 Couchbase server can't handle view query with unlimited number of results
        Environment centos5.6 centos5.6, build 1952 with 16 scheduler
        Description Cluster: 6 nodes
        10.6.2.37
        10.6.2.38
        10.6.2.39
        10.6.2.40
        10.6.2.42
        10.6.2.43

        Build # 2.0.0-1952 with 16 erlang schedulers
        each nodes with 390GB SSD drive, 32GB RAM

        2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.

        Then the UI becomes unresponsive.
        Pay attention to the following stats:

        1st is the erlang scheduler on one of the nodes during query happens:

        (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
        #Fun<erl_eval.6.80247286>
        1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}
        1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}
        1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}
        1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}
        1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}
        1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}
        1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}
        1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}
        1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}
        1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}
        1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}
        1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}
        1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}
        1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}
        1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}
        1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}
        1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}
        1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}
        1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}
        1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}
        1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}
        1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}
        1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}
        1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}
        1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}
        1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}
        1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}
        1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}
        1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}
        1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}
        1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}
        1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}
        1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}
        1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}
        1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}
        1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}
        1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}
        1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}
        1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}
        1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}
        1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}
        1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}

        2nd is the top stats about beam.smp:
          PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
          676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp

        24G memory usage. And the CPU% is always above 350%
        Cluster: 6 nodes
        10.6.2.37
        10.6.2.38
        10.6.2.39
        10.6.2.40
        10.6.2.42
        10.6.2.43

        Build # 2.0.0-1952 with 16 erlang schedulers
        each nodes with 390GB SSD drive, 32GB RAM

        2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.
        I don't put any limit on query results:
            capiUrl = "http://%s:%s/couchBase/" % (cfg.COUCHBASE_IP, cfg.COUCHBASE_PORT)
            url = capiUrl + '%s/_design/%s/_%s/%s' % (bucket,
                                                        design_doc_name, type_,
                                                        view_name)
            headers = {'Content-Type': 'application/json',
                       'Authorization': 'Basic %s' % authorization,
                       'Accept': '*/*'}
             req = urllib2.Request(url, headers = headers)

        Then the UI becomes unresponsive.
        Pay attention to the following stats:

        1st is the erlang scheduler on one of the nodes during query happens:

        (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
        #Fun<erl_eval.6.80247286>
        1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}
        1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}
        1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}
        1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}
        1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}
        1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}
        1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}
        1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}
        1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}
        1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}
        1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}
        1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}
        1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}
        1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}
        1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}
        1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}
        1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}
        1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}
        1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}
        1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}
        1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}
        1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}
        1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}
        1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}
        1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}
        1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}
        1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}
        1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}
        1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}
        1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}
        1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}
        1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}
        1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}
        1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}
        1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}
        1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}
        1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}
        1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}
        1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}
        1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}
        1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}
        1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}

        2nd is the top stats about beam.smp:
          PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
          676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp

        24G memory usage. And the CPU% is always above 350%
        Assignee Chisheng Hong [ chisheng ] Aliaksey Artamonau [ aliaksey artamonau ]
        Chisheng Chisheng Hong (Inactive) made changes -
        Description Cluster: 6 nodes
        10.6.2.37
        10.6.2.38
        10.6.2.39
        10.6.2.40
        10.6.2.42
        10.6.2.43

        Build # 2.0.0-1952 with 16 erlang schedulers
        each nodes with 390GB SSD drive, 32GB RAM

        2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.
        I don't put any limit on query results:
            capiUrl = "http://%s:%s/couchBase/" % (cfg.COUCHBASE_IP, cfg.COUCHBASE_PORT)
            url = capiUrl + '%s/_design/%s/_%s/%s' % (bucket,
                                                        design_doc_name, type_,
                                                        view_name)
            headers = {'Content-Type': 'application/json',
                       'Authorization': 'Basic %s' % authorization,
                       'Accept': '*/*'}
             req = urllib2.Request(url, headers = headers)

        Then the UI becomes unresponsive.
        Pay attention to the following stats:

        1st is the erlang scheduler on one of the nodes during query happens:

        (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
        #Fun<erl_eval.6.80247286>
        1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}
        1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}
        1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}
        1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}
        1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}
        1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}
        1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}
        1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}
        1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}
        1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}
        1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}
        1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}
        1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}
        1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}
        1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}
        1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}
        1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}
        1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}
        1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}
        1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}
        1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}
        1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}
        1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}
        1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}
        1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}
        1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}
        1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}
        1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}
        1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}
        1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}
        1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}
        1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}
        1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}
        1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}
        1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}
        1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}
        1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}
        1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}
        1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}
        1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}
        1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}
        1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}

        2nd is the top stats about beam.smp:
          PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
          676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp

        24G memory usage. And the CPU% is always above 350%
        Cluster: 6 nodes
        10.6.2.37
        10.6.2.38
        10.6.2.39
        10.6.2.40
        10.6.2.42
        10.6.2.43

        Build # 2.0.0-1952 with 16 erlang schedulers
        each nodes with 390GB SSD drive, 32GB RAM

        2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.

        I don't put any limit on query results and those queries are generated without waiting for previous ones finish showing the results:
            capiUrl = "http://%s:%s/couchBase/" % (cfg.COUCHBASE_IP, cfg.COUCHBASE_PORT)
            url = capiUrl + '%s/_design/%s/_%s/%s' % (bucket,
                                                        design_doc_name, type_,
                                                        view_name)
            headers = {'Content-Type': 'application/json',
                       'Authorization': 'Basic %s' % authorization,
                       'Accept': '*/*'}
             req = urllib2.Request(url, headers = headers)

        Then the UI becomes unresponsive.
        Pay attention to the following stats:

        1st is the erlang scheduler on one of the nodes during query happens:

        (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
        #Fun<erl_eval.6.80247286>
        1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}
        1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}
        1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}
        1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}
        1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}
        1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}
        1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}
        1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}
        1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}
        1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}
        1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}
        1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}
        1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}
        1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}
        1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}
        1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}
        1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}
        1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}
        1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}
        1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}
        1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}
        1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}
        1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}
        1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}
        1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}
        1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}
        1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}
        1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}
        1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}
        1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}
        1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}
        1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}
        1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}
        1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}
        1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}
        1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}
        1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}
        1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}
        1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}
        1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}
        1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}
        1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}
        1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}

        2nd is the top stats about beam.smp:
          PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
          676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp

        24G memory usage. And the CPU% is always above 350%
        Chisheng Chisheng Hong (Inactive) made changes -
        Summary Couchbase server can't handle view query with unlimited number of results Couchbase server can't handle hundreds of view queries with unlimited number of results at the same time
        FilipeManana Filipe Manana (Inactive) made changes -
        Assignee Aliaksey Artamonau [ aliaksey artamonau ] Aleksey Kondratenko [ alkondratenko ]
        FilipeManana Filipe Manana (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] Chisheng Hong [ chisheng ]
        steve Steve Yen made changes -
        Labels system-test 2.0-release-notes system-test
        Fix Version/s 2.0.1 [ 10399 ]
        Fix Version/s 2.0 [ 10114 ]
        dipti Dipti Borkar made changes -
        Fix Version/s 2.0.2 [ 10418 ]
        Fix Version/s 2.0.1 [ 10399 ]
        FilipeManana Filipe Manana (Inactive) made changes -
        Attachment erl_crash.dump.tgz [ 15916 ]
        Attachment logs.tgz [ 15917 ]
        FilipeManana Filipe Manana (Inactive) made changes -
        Component/s ns_server [ 10019 ]
        FilipeManana Filipe Manana (Inactive) made changes -
        Assignee Chisheng Hong [ chisheng ] Aliaksey Artamonau [ aliaksey artamonau ]
        jin Jin Lim (Inactive) made changes -
        Fix Version/s 2.1 [ 10414 ]
        jin Jin Lim (Inactive) made changes -
        Planned Start (set to new fixed version's start date)
        Planned End (set to new fixed version's start date)
        maria Maria McDuff (Inactive) made changes -
        Fix Version/s 2.0.2 [ 10418 ]
        maria Maria McDuff (Inactive) made changes -
        Affects Version/s 2.0.1 [ 10399 ]
        Affects Version/s 2.0.2 [ 10418 ]
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aliaksey Artamonau [ aliaksey artamonau ] Ravi Mayuram [ rmayuram ]
        maria Maria McDuff (Inactive) made changes -
        Fix Version/s .next [ 10342 ]
        Fix Version/s 3.0 [ 10414 ]
        maria Maria McDuff (Inactive) made changes -
        Labels 2.0-release-notes system-test 2.0-release-notes scrubbed system-test
        maria Maria McDuff (Inactive) made changes -
        Fix Version/s bug-backlog [ 11600 ]
        Fix Version/s feature-backlog [ 10342 ]
        Triage Untriaged [ 10351 ]
        siri Sriram Melkote made changes -
        Assignee Ravi Mayuram [ rmayuram ] Sriram Melkote [ siri ]
        vmx Volker Mische made changes -
        Status Open [ 1 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            siri Sriram Melkote
            Reporter:
            Chisheng Chisheng Hong (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes