Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35189

ActiveDM trackedWrites iterator positions not maintained after setReplicationTopology [ETA:2019-07-25]

    XMLWordPrintable

Details

    • Triaged
    • No
    • KV-Engine MH Beta part 2

    Description

      Summary
      As seen during a 2 > 4 node rebalance with one replica, and ongoing SyncWrites, the SET_VBUCKET request from ns_server fails with the following error:

      53: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:56583 - 127.0.0.1:11995 (<ud>@ns_server</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":1,"key":"<ud></ud>","keylen":0,"magic":"ClientRequest","opaque":0,"opcode":"SET_VBUCKET","vbucket":15},"refcount":0}] - closing connection ([ 127.0.0.1:56583 - 127.0.0.1:11995 (<ud>@ns_server</ud>) ]): Monotonic<x> (unlabelled) invariant failed: new value (533) breaks invariant on current value (534)

      Steps to Reproduce

      1. Start 4 nodes, 1 replica; initially with 2 nodes in the cluster: 

        COUCHBASE_NUM_VBUCKETS=16 ./cluster_run -n4 --dont-rename
        		 
        # (From a 2nd terminal):
        		 
        cluster_connect -n2 -r1

      2. Add two more nodes (data service only) from UI - http://127.0.0.1:9002 and http://127.0.0.1:9003
      3. Start loading some SyncWrites: 

        cbc-pillowfight -U couchbase://127.0.0.1:12000/default  -u Administrator -P asdasd -d majority -I 50000 -B 1 -m 100 -M 100 -t16 --populate-only

      4. Give pillowfight ~5 seconds, then start a rebalance (from UI).

      Expected Results
      Rebalance completes successfully.

      Actual Results
      Rebalance fails due to the above error.

      Artefacts
      collect_info_MB-35189.zip

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-35275 [Durability]: ADM::highPreparedSeqno throws monotonic exception when ADM moves from null topology to topology with all writes persisted [ETA 2019-07-29]

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-35206 Retry of failed swap rebalance fails with "mover_crashed,\{unexpected_exit,\{\'EXIT\',<0.4022.5>,\{{dcp_wait_for_data_move_failed,"

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-35255 [Durability]: Rebalance out is failing with memcached crash

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-35257 [Durability]: Rebalance (2 nodes In, 1 node Out, Replica change) failed due to memcached crashed.

          • Major - Major loss of function.
          • Closed
          For Gerrit Dashboard: MB-35189
          # Subject Branch Project Status CR V
          112254,1 WIP: MB-35189: Double set_vb_state Monotonic broken master kv_engine Status: ABANDONED 0 -1
          112423,3 MB-35189 [SR]: Transfer node positions on setReplicationTopology master kv_engine Status: MERGED +2 +1
          112697,6 MB-35332 [SR]: Set HPS::it correctly on ActiveDM -> PassiveDM conversion master kv_engine Status: MERGED +2 +1

          Activity

            People

              drigby Dave Rigby (Inactive)
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty