Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6935

"inconsistent_state" in query result after rebalance

    XMLWordPrintable

Details

    • Bug
    • Resolution: Incomplete
    • Major
    • 2.0
    • 2.0-beta
    • ns_server
    • Security Level: Public
    • build-1848

    Description

      Running about 300 queries/sec against 2 buckets

      • Started rebalance
      • Rebalance fails with error in vbucket_mover
        ...
        <0.7975.178> exited with {unexpected_exit,
        {'EXIT',<0.8005.178>,
        {{badmatch,
        [{'EXIT',
        {timeout, {gen_server,call, [<15200.2771.3094>,had_backfill,30000]}

        }}]},
        [

        {ns_single_vbucket_mover, '-wait_backfill_determination/1-fun-1-',1}

        ]}}}

      • during rebalance I see a lot of these errors:

      Stacktrace: [

      {couch_view_merger,simple_set_view_query,3}

      ,

      {couch_httpd,handle_request,6}

      ,

      {mochiweb_http,headers,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]
      [couchdb:error,2012-10-16T8:43:02.960,ns_1@10.6.2.66:<0.4247.177>:couch_log:error:42]Uncaught error in HTTP request: {throw,{error,set_view_outdated}}

      {"error":"error","reason":"inconsistent_state"}

      [couchdb:info,2012-10-16T15:59:14.035,ns_1@10.6.2.66:<0.3334.193>:couch_log:info:39]Set view `default`, group `_design/d1`, missing partitions: [6]
      [couchdb:error,2012-10-16T15:59:19.043,ns_1@10.6.2.66:<0.7960.182>:couch_log:error:42]Uncaught error in HTTP request: {throw,{error,inconsistent_state}}

      • cbstats shows that partition [6] is neither active nor replica on10.6.2.66,

      [root@pine-11802 logs]# /opt/couchbase/bin/cbstats localhost:11210 vbucket -b saslbucket -p password | grep vb_6:
      [root@pine-11802 logs]#

      *however the file exists in its data directory (could be we failed to move?)
      /opt/couchbase/bin/couch_dbinfo /data/default/6.couch.12
      DB Info (/data/default/6.couch.12)
      file format version: 10
      update_seq: 20614
      doc count: 13566
      deleted doc count: 112
      data size: 13.16 MB
      B-tree size: 1.25 MB
      total disk size: 18.18 MB

      *indexer doesn't expect this db to be here either
      curl -s -u Administrator:password 'http://10.6.2.66:8092/_set_view/default/_design/d1/_info' | json_xs
      "active_partitions" : [
      0,
      1,
      2,
      3,
      4,
      5,
      7,
      8,
      9,

      (full logs to be attached)

      Attachments

        1. 66node_vbstats.txt
          8 kB
        2. 67node_vbstats.txt
          8 kB
        3. 68node_vbstats.txt
          9 kB
        4. 69node_vbstats.txt
          9 kB
        5. node66_info.json
          23 kB
        6. node67_info.json
          23 kB
        7. node68_info.json
          23 kB
        8. node69_info.json
          23 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            tommie Tommie McAfee (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty