Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-15729

[Windows] Build index is hung in test test_rebalance_in (rebalance in of KV node) in job add-index

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 4.0.0
    • 4.0.0
    • secondary-index
    • Security Level: Public
    • Windows
    • Untriaged
    • Yes

    Description

      Run - http://qa.sc.couchbase.com/job/w012R2-2i-vset03-03-recoverytests-add-index/19/consoleFull

      Test case: test_1

      ./testrunner -i /tmp/recoverytests-add-index.ini doc-per-day=50,standard_buckets=1,sasl_buckets=1,skip_build_tuq=True,cbq_version=sherlock,scan_consistency=request_plus -t 2i.recovery_2i.SecondaryIndexingRecoveryTests.test_rebalance_in,initial=,before=create_index,after=query_ops,groups=simple,dataset=default,doc-per-day=10,nodes_init=5,nodes_in=1,services_in=kv,services_init=n1ql:kv-kv-kv-index-index,GROUP=REB-IN;P1

      The build of index employee40d14b2f41f744a5b85637dc7724f198join_yr does not complete in 10 mins after which the test fails.

      Test steps:

      1. Create 3 buckets, load 100800 documents
      2. Create a bunch of indexes (and build them fully)
      3. Create a few more indexes including employee40d14b2f41f744a5b85637dc7724f198join_yr and start build of the indexes
      4. While indexes are building, perform rebalance in of KV node:
      adding remote node @172.23.105.88:8091 to this cluster @172.23.105.191:8091

      Build index of employee40d14b2f41f744a5b85637dc7724f198join_yr is hung. Queries executed and their timestamps are:

      2015-07-16 19:18:17 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX employee40d14b2f41f744a5b85637dc7724f198join_yr ON bucket0(join_yr) USING GSI WITH

      {'defer_build': True}

      2015-07-16 19:18:17 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+employee40d14b2f41f744a5b85637dc7724f198join_yr+ON+bucket0%28join_yr%29+USING+GSI+WITH%7B%27defer_build%27%3A+True%7D
      2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 353.0029ms
      2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX employee40d14b2f41f744a5b85637dc7724f198join_yr ON default(join_yr) USING GSI WITH

      {'defer_build': True}

      2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+employee40d14b2f41f744a5b85637dc7724f198join_yr+ON+default%28join_yr%29+USING+GSI+WITH%7B%27defer_build%27%3A+True%7D
      2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 213.0406ms
      2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX employee40d14b2f41f744a5b85637dc7724f198join_yr ON standard_bucket0(join_yr) USING GSI WITH

      {'defer_build': True}

      2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+employee40d14b2f41f744a5b85637dc7724f198join_yr+ON+standard_bucket0%28join_yr%29+USING+GSI+WITH%7B%27defer_build%27%3A+True%7D
      2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 307.9907ms
      2015-07-16 19:18:20 | INFO | MainProcess | test_thread | [base_2i.async_build_index] BUILD INDEX on bucket0(employee40d14b2f41f744a5b85637dc7724f198job_title,employee40d14b2f41f744a5b85637dc7724f198join_yr) USING GSI
      2015-07-16 19:18:20 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on bucket0(employee40d14b2f41f744a5b85637dc7724f198job_title,employee40d14b2f41f744a5b85637dc7724f198join_yr) USING GSI
      2015-07-16 19:18:20 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=BUILD+INDEX+on+bucket0%28employee40d14b2f41f744a5b85637dc7724f198job_title%2Cemployee40d14b2f41f744a5b85637dc7724f198join_yr%29+USING+GSI
      2015-07-16 19:18:21 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 475.0493ms
      2015-07-16 19:18:21 | INFO | MainProcess | test_thread | [base_2i.async_build_index] BUILD INDEX on default(employee40d14b2f41f744a5b85637dc7724f198job_title,employee40d14b2f41f744a5b85637dc7724f198join_yr) USING GSI
      2015-07-16 19:18:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee40d14b2f41f744a5b85637dc7724f198job_title,employee40d14b2f41f744a5b85637dc7724f198join_yr) USING GSI
      2015-07-16 19:18:22 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee40d14b2f41f744a5b85637dc7724f198job_title%2Cemployee40d14b2f41f744a5b85637dc7724f198join_yr%29+USING+GSI
      2015-07-16 19:18:43 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 21.583002s
      2015-07-16 19:18:43 | INFO | MainProcess | test_thread | [base_2i.async_build_index] BUILD INDEX on standard_bucket0(employee40d14b2f41f744a5b85637dc7724f198job_title,employee40d14b2f41f744a5b85637dc7724f198join_yr) USING GSI
      2015-07-16 19:18:44 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on standard_bucket0(employee40d14b2f41f744a5b85637dc7724f198job_title,employee40d14b2f41f744a5b85637dc7724f198join_yr) USING GSI
      2015-07-16 19:18:44 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=BUILD+INDEX+on+standard_bucket0%28employee40d14b2f41f744a5b85637dc7724f198job_title%2Cemployee40d14b2f41f744a5b85637dc7724f198join_yr%29+USING+GSI
      2015-07-16 19:19:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 27.0490009s

      Logs:
      https://s3-ap-southeast-1.amazonaws.com/indexing-test/addIndex_Windows_19/collectinfo-2015-07-17T085846-ns_1%40172.23.105.191.zip
      https://s3-ap-southeast-1.amazonaws.com/indexing-test/addIndex_Windows_19/collectinfo-2015-07-17T085846-ns_1%40172.23.105.240.zip
      https://s3-ap-southeast-1.amazonaws.com/indexing-test/addIndex_Windows_19/collectinfo-2015-07-17T085846-ns_1%40172.23.105.91.zip
      https://s3-ap-southeast-1.amazonaws.com/indexing-test/addIndex_Windows_19/collectinfo-2015-07-17T085846-ns_1%40172.23.107.144.zip
      https://s3-ap-southeast-1.amazonaws.com/indexing-test/addIndex_Windows_19/collectinfo-2015-07-17T085846-ns_1%40172.23.107.145.zip

      Cluster config:

      [SERVER_1]
      ip:172.23.105.191
      services=n1ql,kv

      [SERVER_2]
      ip:172.23.105.240
      services=kv

      [SERVER_3]
      ip:172.23.107.144
      services=kv

      [SERVER_4]
      ip:172.23.107.145
      services=index

      [SERVER_5]
      ip:172.23.105.91
      services=index

      [SERVER_6]
      ip:172.23.105.88
      services=kv

      Note: This test does not fail on Centos

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            siri Sriram Melkote (Inactive)
            prathibha Prathibha Bisarahalli (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty