Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6271

Index compaction termination was seen in the logs after rebalacing in nodes

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Major
    • 2.0
    • 2.0
    • view-engine
    • Security Level: Public
    • None
    • build #1598 on Ubuntu 64bit

    Description

      Index compaction is stuck after rebalacing in nodes. I did the following steps:

      1. Create sasl bucket with password(build#1598).
      2. Load 1M items
      3. Create 11 production ddocs(2 views per ddoc)

      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d1' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.key_num);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,doc.key);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d2' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.name);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.key_num]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d3' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.email);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.name]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d4' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.city);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.email]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d5' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.country);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.city]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d6' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.realm);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.country]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d7' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.coins);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.realm]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d8' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.category);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.coins]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d9' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.achievements);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.category]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d10' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.body);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.achievements]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d11' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.city+doc.country);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.body]);\n}

      "} }}'

      4. While the initial indexing is in progress, add 3 nodes and rebalance.

      5. The index compaction gets stuck for one of the design docs. Following error can be seen in the diag repeatedly. After a while it filled up the disk space on the VM.

      [ns_server:info,2012-08-17T4:19:44.465,ns_1@10.1.3.67:<0.29908.4>:compaction_daemon:spawn_view_index_compactor:664]Compacting indexes for saslbucket/_design/d9/main
      [couchdb:info,2012-08-17T4:19:44.608,ns_1@10.1.3.67:<0.20718.0>:couch_log:info:39]Starting updater for set view `saslbucket`, main group `_design/d9`
      [couchdb:info,2012-08-17T4:19:44.609,ns_1@10.1.3.67:<0.29910.4>:couch_log:info:39]Updater for set view `saslbucket`, main group `_design/d9` started
      Active partitions: [32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,56,57,58,59,60,61,62,63]
      Passive partitions: []
      Cleanup partitions: [55,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,106,107,108,109,110,111,112,113,114,115,116,127]
      Replicas to transfer: []
      Pending transition:
      active: [55]
      passive: []
      Initial build: false

      [couchdb:info,2012-08-17T4:19:44.610,ns_1@10.1.3.67:<0.29916.4>:couch_log:info:39]Updater reading changes from active partitions to update main set view group `_design/d9` from set `saslbucket`
      [couchdb:info,2012-08-17T4:19:48.180,ns_1@10.1.3.67:<0.29915.4>:couch_log:info:39]Updater for set view `saslbucket`, main group `_design/d9`, performed cleanup of 118326 key/value pairs in 3.438 seconds
      [couchdb:info,2012-08-17T4:19:48.184,ns_1@10.1.3.67:<0.29915.4>:couch_log:info:39]Updater checkpointing set view `saslbucket` update for main group `_design/d9`
      [couchdb:info,2012-08-17T4:19:48.188,ns_1@10.1.3.67:<0.20718.0>:couch_log:info:39]Stopping updater for set view `saslbucket`, main group `_design/d9`
      [couchdb:info,2012-08-17T4:19:48.192,ns_1@10.1.3.67:<0.20718.0>:couch_log:info:39]Updater, set view `saslbucket`, main group `_design/d9`, stopped with reason:

      {updater_error, shutdown}

      [couchdb:info,2012-08-17T4:19:48.195,ns_1@10.1.3.67:<0.20718.0>:couch_log:info:39]Restarting compaction for main group `_design/d9`, set view `saslbucket`. Reason: partition states were updated
      [ns_server:warn,2012-08-17T4:19:48.195,ns_1@10.1.3.67:<0.29907.4>:compaction_daemon:do_chain_compactors:511]Compactor for view `saslbucket/_design/d9/main` (pid [

      {type,view},
      {important,true},
      {name, <<"saslbucket/_design/d9/main">>},
      {fa,
      {#Fun<compaction_daemon.16.13428321>,
      [<<"saslbucket">>,
      <<"_design/d9">>,
      main,
      {config,
      {10, 18446744073709551616},
      {10, 18446744073709551616},
      undefined,false,
      {daemon_config,30,
      131072}},
      false,bucket]}}]) terminated unexpectedly: shutdown
      [ns_server:warn,2012-08-17T4:19:48.200,ns_1@10.1.3.67:<0.29880.4>:compaction_daemon:do_chain_compactors:516]Compactor for view `saslbucket/_design/d9` (pid [{type,view}

      ,

      {name, <<"saslbucket/_design/d9">>}

      ,

      {important,false}

      ,
      {fa,
      {#Fun<compaction_daemon.21.91655267>,
      [<<"saslbucket">>,
      <<"_design/d9">>,
      {config,

      {10,18446744073709551616},
      {10,18446744073709551616}

      ,
      undefined,false,
      {daemon_config,30,
      131072}},
      false,bucket]}}]) terminated unexpectedly (ignoring this): shutdown
      [ns_server:debug,2012-08-17T4:19:48.201,ns_1@10.1.3.67:compaction_daemon:compaction_daemon:handle_info:306]Finished compaction iteration.
      [ns_server:debug,2012-08-17T4:19:48.203,ns_1@10.1.3.67:compaction_daemon:compaction_daemon:schedule_next_compaction:1129]Finished compaction too soon. Next run will be in 26s

      Diagnostics are attached.

      Attachments

        1. 10.1.3.227-8091-diag.txt.gz
          3.26 MB
        2. 10.1.3.67-8091-diag.txt.gz
          3.83 MB
        3. 10.1.3.70-8091-diag.txt.gz
          2.71 MB
        4. 10.1.3.73-8091-diag.txt.gz
          2.38 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            farshid Farshid Ghods (Inactive)
            deepkaran.salooja Deepkaran Salooja
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty