Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10031

couch_view_index_updater process gets stuck seemingly while running make simple-test

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Blocker
    • 3.0
    • 3.0
    • view-engine
    • Security Level: Public
    • None

    Description

      It's not happening always. Perhaps even rare. But here's what I see in testrunner output:

      0}: Generated results contains 500 items
      [2014-01-27 14:55:36,759] - [rest_client:431] INFO - index query url: http://10.17.15.219:9500/default/_design/test_view-6bf647d/_view/test_view-6bf647d?stale=false&connection_timeout=60000
      [2014-01-27 14:56:34,856] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 14:57:34,857] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 14:58:34,857] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 14:59:34,857] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:00:34,857] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:01:34,858] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:02:34,858] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:03:34,858] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:04:34,858] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:05:34,859] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:06:34,859] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:07:34,859] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:08:34,859] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:09:34,860] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:10:34,860] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:10:36,856] - [rest_client:689] ERROR - socket error while connecting to http://10.17.15.219:9500/default/_design/test_view-6bf647d/_view/test_view-6bf647d?stale=false&connection_timeout=60000 error timed out
      [2014-01-27 15:10:36,856] - [task:1611] ERROR - view_results not ready yet ddoc=test_view-6bf647d , try again in 10 seconds...
      [2014-01-27 15:10:46,866] - [rest_client:431] INFO - index query url: http://10.17.15.219:9500/default/_design/test_view-6bf647d/_view/test_view-6bf647d?stale=false&connection_timeout=60000
      [2014-01-27 15:11:34,860] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:12:34,860] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:13:34,861] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:14:34,861] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:15:34,861] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:16:34,861] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:17:34,862] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:18:34,862] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:19:34,862] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:20:34,863] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:21:34,863] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:22:34,863] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:23:34,863] - [viewquerytests:2027] INFO - Current amount of threads 1
      [2014-01-27 15:24:34,864] - [viewquerytests:2027] INFO - Current amount of threads 1

      When I open UI and do view query it works. But it gets stuck when I do stale=false query.

      ps also has indexer process despite no load on system for tens of minutes (I went for lunch just after starting make simple-test). I.e.:

      root 18895 0.0 0.0 5940 896 pts/0 S+ 14:50 0:00 | _ make simple-test
      root 18896 0.0 0.0 10992 1540 pts/0 S+ 14:50 0:00 | _ /bin/bash scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr.ini conf/simple.conf
      root 19182 0.0 0.0 34396 6996 pts/0 S+ 14:50 0:00 | _ python ./cluster_run --nodes=4
      root 19229 0.0 0.1 1685344 32904 pts/0 Sl 14:50 0:01 | | _ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -MMmcs30 -A 16 -sbtu -P 327680 – -root /usr/lib/erlang -progname erl – -home /root – -pa /root/src/repo3
      root 19419 3.6 0.8 1952068 268844 ? SLsl 14:50 1:28 | | | _ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -sbt u -P 327680 -K true -MMmcs 30 – -root /usr/lib/erlang -progname erl – -home /root – -smp
      root 19511 0.0 0.0 4324 672 ? Ss 14:50 0:00 | | | | _ sh -s disksup
      root 19516 0.0 0.0 4208 596 ? Ss 14:50 0:00 | | | | _ /usr/lib/erlang/lib/os_mon-2.2.14/priv/bin/memsup
      root 19520 0.0 0.0 4208 352 ? Ss 14:50 0:00 | | | | _ /usr/lib/erlang/lib/os_mon-2.2.14/priv/bin/cpu_sup
      root 19528 0.0 0.0 11432 444 ? Ss 14:50 0:00 | | | | _ inet_gethost 4
      root 19529 0.0 0.0 11432 452 ? S 14:50 0:00 | | | | | _ inet_gethost 4
      root 20050 0.0 0.0 11432 448 ? S 14:51 0:00 | | | | | _ inet_gethost 4
      root 19535 0.0 0.0 794096 1504 ? Ssl 14:50 0:00 | | | | _ /root/src/repo30/ns_server/priv/i386-linux-godu
      root 19703 0.2 0.0 8588 748 ? Ss 14:50 0:06 | | | | _ portsigar for n_0@127.0.0.1
      root 21616 0.0 0.0 64112 2944 ? Ss 14:55 0:00 | | | | _ /root/src/repo30/install/bin/couch_view_index_updater
      root 19682 0.0 0.0 1288836 26584 ? Ssl 14:50 0:00 | | | _ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -P 327680 -K true – -root /usr/lib/erlang -progname erl – -home /root – -smp enable -kernel error_lo
      root 19840 0.0 0.0 11432 440 ? Ss 14:51 0:00 | | | _ inet_gethost 4
      root 19841 0.0 0.0 11432 400 ? S 14:51 0:00 | | | | _ inet_gethost 4
      root 20357 11.3 0.2 279056 88636 ? Ssl 14:52 4:27 | | | _ /root/src/repo30/install/bin/memcached.orig -u root -X /root/src/repo30/install/lib/memcached/stdin_term_handler.so -X /root/src/repo30/install/
      root 21495 0.0 0.0 10980 1376 ? Ss 14:54 0:00 | | | _ /bin/bash /root/src/repo30/install/bin/moxi -Z port_listen=12001,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_m
      root 21500 0.0 0.0 177172 4580 ? Sl 14:54 0:00 | | | _ /root/src/repo30/install/bin/moxi.orig -u root -Z port_listen=12001,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,co
      root 21501 0.0 0.0 5804 672 ? S 14:54 0:00 | | | _ tee /root/src/repo30/install/../moxi.21501.log
      root 19231 0.0 0.0 1680696 30756 pts/0 Sl 14:50 0:00 | | _ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -MMmcs30 -A 16 -sbtu -P 327680 – -root /usr/lib/erlang -progname erl – -home /root – -pa /root/src/repo3
      root 19432 1.8 0.5 1881960 195488 ? SLsl 14:50 0:44 | | | _ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -sbt u -P 327680 -K true -MMmcs 30 – -root /usr/lib/erlang -progname erl – -home /root – -smp
      root 19509 0.0 0.0 4324 672 ? Ss 14:50 0:00 | | | | _ sh -s disksup
      root 19517 0.0 0.0 4208 596 ? Ss 14:50 0:00 | | | | _ /usr/lib/erlang/lib/os_mon-2.2.14/priv/bin/memsup
      root 19519 0.0 0.0 4208 352 ? Ss 14:50 0:00 | | | | _ /usr/lib/erlang/lib/os_mon-2.2.14/priv/bin/cpu_sup
      root 19532 0.0 0.0 11432 440 ? Ss 14:50 0:00 | | | | _ inet_gethost 4
      root 19533 0.0 0.0 11432 452 ? S 14:50 0:00 | | | | | _ inet_gethost 4
      root 21054 0.0 0.0 11432 452 ? S 14:54 0:00 | | | | | _ inet_gethost 4
      root 21517 0.0 0.0 793456 1032 ? Ssl 14:54 0:00 | | | | _ /root/src/repo30/ns_server/priv/i386-linux-godu
      root 21530 0.2 0.0 8588 748 ? Ss 14:54 0:05 | | | | _ portsigar for n_1@127.0.0.1
      root 21525 0.0 0.0 1290116 27520 ? Ssl 14:54 0:00 | | | _ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -P 327680 -K true – -root /usr/lib/erlang -progname erl – -home /root – -smp enable -kernel error_lo
      root 21566 0.0 0.1 149884 59256 ? Ssl 14:54 0:00 | | | _ /root/src/repo30/install/bin/memcached.orig -u root -X /root/src/repo30/install/lib/memcached/stdin_term_handler.so -X /root/src/repo30/install/
      root 21578 0.0 0.0 10980 1376 ? Ss 14:55 0:00 | | | _ /bin/bash /root/src/repo30/install/bin/moxi -Z port_listen=12003,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_m
      root 21579 0.0 0.0 177040 4320 ? Sl 14:55 0:00 | | | _ /root/src/repo30/install/bin/moxi.orig -u root -Z port_listen=12003,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,co
      root 21580 0.0 0.0 5804 664 ? S 14:55 0:00 | | | _ tee /root/src/repo30/install/../moxi.21580.log
      root 19236 0.0 0.0 1678496 29308 pts/0 Sl 14:50 0:00 | | _ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -MMmcs30 -A 16 -sbtu -P 327680 – -root /usr/lib/erlang -progname erl – -home /root – -pa /root/src/repo3
      root 19378 1.3 0.5 1795844 172360 ? SLsl 14:50 0:33 | | | _ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -sbt u -P 327680 -K true -MMmcs 30 – -root /usr/lib/erlang -progname erl – -home /root – -smp
      root 19521 0.0 0.0 4324 668 ? Ss 14:50 0:00 | | | | _ sh -s disksup
      root 19523 0.0 0.0 4208 600 ? Ss 14:50 0:00 | | | | _ /usr/lib/erlang/lib/os_mon-2.2.14/priv/bin/memsup
      root 19524 0.0 0.0 4208 348 ? Ss 14:50 0:00 | | | | _ /usr/lib/erlang/lib/os_mon-2.2.14/priv/bin/cpu_sup
      root 19544 0.0 0.0 11432 440 ? Ss 14:50 0:00 | | | | _ inet_gethost 4
      root 19545 0.0 0.0 11432 444 ? S 14:50 0:00 | | | | | _ inet_gethost 4
      root 20133 0.0 0.0 793456 1032 ? Ssl 14:51 0:00 | | | | _ /root/src/repo30/ns_server/priv/i386-linux-godu
      root 20248 0.2 0.0 8588 744 ? Ss 14:51 0:05 | | | | _ portsigar for n_2@127.0.0.1
      root 20244 0.0 0.0 1290116 27932 ? Ssl 14:51 0:00 | | | _ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -P 327680 -K true – -root /usr/lib/erlang -progname erl – -home /root – -smp enable -kernel error_lo
      root 20285 0.0 0.1 149884 59256 ? Ssl 14:51 0:00 | | | _ /root/src/repo30/install/bin/memcached.orig -u root -X /root/src/repo30/install/lib/memcached/stdin_term_handler.so -X /root/src/repo30/install/
      root 20492 0.0 0.0 10980 1380 ? Ss 14:52 0:00 | | | _ /bin/bash /root/src/repo30/install/bin/moxi -Z port_listen=12005,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_m
      root 20493 0.0 0.0 177040 4320 ? Sl 14:52 0:00 | | | _ /root/src/repo30/install/bin/moxi.orig -u root -Z port_listen=12005,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,co
      root 20494 0.0 0.0 5804 668 ? S 14:52 0:00 | | | _ tee /root/src/repo30/install/../moxi.20494.log
      root 19245 0.0 0.0 1678828 30072 pts/0 Sl 14:50 0:00 | | _ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -MMmcs30 -A 16 -sbtu -P 327680 – -root /usr/lib/erlang -progname erl – -home /root – -pa /root/src/repo3
      root 19373 1.3 0.5 1795736 172100 ? SLsl 14:50 0:32 | | _ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -sbt u -P 327680 -K true -MMmcs 30 – -root /usr/lib/erlang -progname erl – -home /root – -smp
      root 19510 0.0 0.0 4324 668 ? Ss 14:50 0:00 | | | _ sh -s disksup
      root 19515 0.0 0.0 4208 596 ? Ss 14:50 0:00 | | | _ /usr/lib/erlang/lib/os_mon-2.2.14/priv/bin/memsup
      root 19518 0.0 0.0 4208 352 ? Ss 14:50 0:00 | | | _ /usr/lib/erlang/lib/os_mon-2.2.14/priv/bin/cpu_sup
      root 19530 0.0 0.0 11432 440 ? Ss 14:50 0:00 | | | _ inet_gethost 4
      root 19531 0.0 0.0 11432 444 ? S 14:50 0:00 | | | | _ inet_gethost 4
      root 20129 0.0 0.0 793456 1032 ? Ssl 14:51 0:00 | | | _ /root/src/repo30/ns_server/priv/i386-linux-godu
      root 20208 0.2 0.0 8588 748 ? Ss 14:51 0:05 | | | _ portsigar for n_3@127.0.0.1
      root 20204 0.0 0.0 1290116 27608 ? Ssl 14:51 0:00 | | _ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -P 327680 -K true – -root /usr/lib/erlang -progname erl – -home /root – -smp enable -kernel error_lo
      root 20273 0.0 0.1 149888 59256 ? Ssl 14:51 0:00 | | _ /root/src/repo30/install/bin/memcached.orig -u root -X /root/src/repo30/install/lib/memcached/stdin_term_handler.so -X /root/src/repo30/install/
      root 20503 0.0 0.0 10980 1376 ? Ss 14:52 0:00 | | _ /bin/bash /root/src/repo30/install/bin/moxi -Z port_listen=12007,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_m
      root 20504 0.0 0.0 177040 4320 ? Sl 14:52 0:00 | | _ /root/src/repo30/install/bin/moxi.orig -u root -Z port_listen=12007,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,co
      root 20505 0.0 0.0 5804 668 ? S 14:52 0:00 | | _ tee /root/src/repo30/install/../moxi.20505.log
      root 19183 0.0 0.0 74716 20272 pts/0 S+ 14:50 0:00 | _ python ./testrunner -c conf/simple.conf -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True
      root 19222 1.3 0.1 554760 57080 pts/0 Sl+ 14:50 0:32 | | _ python ./testrunner -c conf/simple.conf -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True
      root 19184 0.0 0.0 5804 664 pts/0 S+ 14:50 0:00 | _ tee make_test.log

      Attaching /diag.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            FilipeManana Filipe Manana (Inactive)
            alkondratenko Aleksey Kondratenko (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty