Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-22494

[FTS] Indexing is stuck while a restarted node is loading its dataDir

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.0.0
    • 5.0.0
    • fts
    • None

    Description

      Build
      5.0.0-1755

      Testcase(new)
      ./testrunner -i INI_FILE.ini -t fts.moving_topology_fts.MovingTopFTS.update_index_during_failover,items=200000,cluster=D+F:F,skip-cleanup=True

      Steps
      1. Cluster : [kv(.139), kv+fts(.175), fts(.176)]
      2. Load 200k items and wait for indexing to cross the half-item mark. Indexing is stuck at 50%.

      Was simply playing with node service combinations, trying to control the time of failover but before I even got to the failover phase, indexing got stuck at 50% for atleast 4 mins. Im not sure if this is going to be consistently reproducible so I collected the logs from this run.

      2017-01-26 10:52:10 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 1554
      		 
      2017-01-26 10:52:17 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 8474
      		 
      2017-01-26 10:52:24 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 17078
      		 
      2017-01-26 10:52:30 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 25190
      		 
      2017-01-26 10:52:37 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 33128
      		 
      2017-01-26 10:52:44 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 43666
      		 
      2017-01-26 10:52:50 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 53652
      		 
      2017-01-26 10:52:57 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 62115
      		 
      2017-01-26 10:53:04 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 71229
      		 
      2017-01-26 10:53:12 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 82701
      		 
      2017-01-26 10:53:18 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 90539
      		 
      2017-01-26 10:53:25 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 97844
      		 
      2017-01-26 10:53:31 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:53:38 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:53:44 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:53:51 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:53:57 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:54:03 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:54:10 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:54:16 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:54:23 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:54:29 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:54:36 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:54:42 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:54:49 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:54:55 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:55:02 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:55:08 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:55:15 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000
      		 
      2017-01-26 10:55:21 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 200000, docs in FTS index 'default_index_1': 100000

      From logs on.176, I see no errors but indexing is slow (1.5 mins to index 10k docs or so).

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-45375 CBGT loadDataDir and janitor operations overlap

          • Major - Major loss of function.
          • Open
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              apiravi Aruna Piravi (Inactive)
              apiravi Aruna Piravi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty