Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59759

ERROR 257: Monotonic<m> (PassiveStream(vb:871 eq_dcpq:replication:ns_1@172.23.216.146->ns_1@172.23.216.142:default)::cur_snapshot_start) invariant failed: new value (1114) breaks invariant on current value (1120).

    XMLWordPrintable

Details

    Description

      Steps To Recreate:

      1. Create a 4 node cluster
      2. Create a magma bucket(replicas =2)
      3. Load 100000 docs(doc_size = 256 bytes)
      4. While data loading is going on failover a node and rebalance afterwards
      5. Add back the above node
      6. Observed ERROR 257: Exception occurred during packet execution. Closing connection [ {"ip":"127.0.0.1","port":48964}

        -

        {"ip":"127.0.0.1","port":11209}

        (System, @ns_server) ]: Monotonic<m> (PassiveStream(vb:871 eq_dcpq:replication:ns_1@172.23.216.146->ns_1@172.23.216.142:default)::cur_snapshot_start) invariant failed: new value (1114) breaks invariant on current value (1120).

      Below error is from node 172.23.216.142

      Error:

      2023-11-21T03:59:14.138587+00:00 ERROR 257: Exception occurred during packet execution. Closing connection [ {"ip":"127.0.0.1","port":48964} - {"ip":"127.0.0.1","port":11209} (System, @ns_server) ]: Monotonic<m> (PassiveStream(vb:871 eq_dcpq:replication:ns_1@172.23.216.146->ns_1@172.23.216.142:default)::cur_snapshot_start) invariant failed: new value (1114) breaks invariant on current value (1120). Cookies: [{"aiostat":"success","ewouldblock":false,"packet":{"bodylen":37,"cas":0,"datatype":"raw","extlen":1,"extras":{"version":0},"keylen":0,"magic":"ClientRequest","opaque":254,"opcode":"DCP_SNAPSHOT_MARKER","value":{"end_seqno":"1128","flags":["raw: 0x00000005","Memory","Checkpoint"],"high_completed_seqno":"0","max_visible_seqno":"1128","start_seqno":"1114"},"vbucket":871},"refcount":1,"started":"1002862086048103 (207 us ago)","throttled":false}] Exception thrown from: ["#0  /opt/couchbase/bin/memcached() [0x400000+0x144364]","#1  /opt/couchbase/bin/memcached() [0x400000+0x5e8bb5]","#2  /opt/couchbase/bin/memcached() [0x400000+0x5ea3d1]","#3  /opt/couchbase/bin/memcached() [0x400000+0x5ee956]","#4  /opt/couchbase/bin/memcached() [0x400000+0x5dc8ac]","#5  /opt/couchbase/bin/memcached() [0x400000+0x5dc9ed]","#6  /opt/couchbase/bin/memcached() [0x400000+0x3419e0]","#7  /opt/couchbase/bin/memcached() [0x400000+0x1dccb3]","#8  /opt/couchbase/bin/memcached() [0x400000+0x2a8a89]","#9  /opt/couchbase/bin/memcached() [0x400000+0x228f0b]","#10 /opt/couchbase/bin/memcached() [0x400000+0x20f8c0]","#11 /opt/couchbase/bin/memcached() [0x400000+0x2147b4]","#12 /opt/couchbase/bin/memcached() [0x400000+0x21e9d7]","#13 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f0ee2ef4000+0xf84e]","#14 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f0ee2ef4000+0x18b99]","#15 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(event_base_loop+0x357) [0x7f0ee2ef4000+0x19287]","#16 /opt/couchbase/bin/memcached() [0x400000+0x97bcc6]","#17 /opt/couchbase/bin/memcached() [0x400000+0x97c1ce]","#18 /opt/couchbase/bin/memcached() [0x400000+0x97e698]","#19 /opt/couchbase/bin/memcached() [0x400000+0x1cf0d9]","#20 /opt/couchbase/bin/memcached() [0x400000+0x8ce91c]","#21 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f0ee213f000+0xe4aa3]","#22 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f0ee289c000+0x7ea7]","#23 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f0ee1e00000+0xfba2f]"]
      

      QE-TEST:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /data/workspace/debian-p0-durability-vset00-00-tunable_rebalance_swap_majority_6.5_P0/testexec.10069.ini process_concurrency=8,sdk_timeout=60,num_items=100000,doc_size=256,GROUP=P0;durability,EXCLUDE_GROUP=non_dgm;not_for_majority,durability=MAJORITY,active_resident_threshold=70,get-cbcollect-info=True,infra_log_level=info,log_level=info,upgrade_version=7.6.0-1813,sirius_url=http://172.23.120.103:4000 -t rebalance_new.swaprebalancetests.SwapRebalanceFailedTests.test_add_back_failed_node,nodes_init=4,replicas=2,standard_buckets=1,num-swap=1,swap-orchestrator=True,GROUP=P0;durability'
      

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-59759
          # Subject Branch Project Status CR V

          Activity

            People

              ankush.sharma Ankush Sharma
              ankush.sharma Ankush Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty