Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46090

Couchstore: ThrowExceptionUnderflowPolicy taking down DCP connection

    XMLWordPrintable

Details

    Description

      1. Create a 3 node cluster
      2. Create required buckets and collections.
      3. Create 100000000 items sequentially
      4. Rebalance in with Loading of docs
      5. Rebalance Out with Loading of docs
      6. Rebalance In_Out with Loading of docs
      7. Swap with Loading of docs
      8. Failover a node and RebalanceOut that node with loading in parallel

      Seeing the below message in the logs during last rebalance operation.

      Node 172.23.121.124

      '2021-05-03T11:24:52.288413-07:00 ERROR 726: Exception occurred during packet execution. Closing connection: ThrowExceptionUnderflowPolicy current:9223368318523677506 arg:8244557403892. Cookies: [{"aiostat":"success","connection":"[ {\\"ip\\":\\"127.0.0.1\\",\\"port\\":35209} - {\\"ip\\":\\"127.0.0.1\\",\\"port\\":11209} (System, <ud>@ns_server</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":39,"cas":1620066281792274436,"datatype":"raw","extlen":20,"key":"<ud>.random-00040969104</ud>","keylen":19,"magic":"ClientRequest","opaque":347,"opcode":"DCP_EXPIRATION","vbucket":845},"refcount":1}] Exception thrown from: ["#0  /opt/couchbase/bin/memcached() [0x400000+0x2c7543]","#1  /opt/couchbase/bin/memcached() [0x400000+0x8c076]","#2  /opt/couchbase/bin/memcached() [0x400000+0x2fd3b7]","#3  /opt/couchbase/bin/memcached() [0x400000+0x3a08df]","#4  /opt/couchbase/bin/memcached() [0x400000+0x2f522b]","#5  /opt/couchbase/bin/memcached() [0x400000+0x2f560c]","#6  /opt/couchbase/bin/memcached() [0x400000+0x3ff932]","#7  /opt/couchbase/bin/memcached() [0x400000+0x2f2469]","#8  /opt/couchbase/bin/memcached() [0x400000+0x2f3601]","#9  /opt/couchbase/bin/memcached() [0x400000+0x302e1f]","#10 /opt/couchbase/bin/memcached() [0x400000+0x2b9ee8]","#11 /opt/couchbase/bin/memcached() [0x400000+0x4a6a21]","#12 /opt/couchbase/bin/memcached() [0x400000+0x4a7db1]","#13 /opt/couchbase/bin/memcached() [0x400000+0x49b446]","#14 /opt/couchbase/bin/memcached() [0x400000+0x49b683]","#15 /opt/couchbase/bin/memcached() [0x400000+0x49b80a]","#16 /opt/couchbase/bin/memcached() [0x400000+0x25705f]","#17 /opt/couchbase/bin/memcached() [0x400000+0x16c5e4]","#18 /opt/couchbase/bin/memcached() [0x400000+0x1f74a9]","#19 /opt/couchbase/bin/memcached() [0x400000+0x1cbb98]","#20 /opt/couchbase/bin/memcached() [0x400000+0x1b0f48]","#21 /opt/couchbase/bin/memcached() [0x400000+0x1b2777]","#22 /opt/couchbase/bin/memcached() [0x400000+0x1b3391]","#23 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f1954be1000+0xed8e]","#24 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f1954be1000+0x17d01]","#25 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(event_base_loop+0x3bf) [0x7f1954be1000+0x1854f]","#26 /opt/couchbase/bin/memcached() [0x400000+0x18cc3b]","#27 /opt/couchbase/bin/memcached() [0x400000+0x6b0609]","#28 /lib64/libpthread.so.0() [0x7f19556f3000+0x7ea5]","#29 /lib64/libc.so.6(clone+0x6d) [0x7f1953161000+0xfe8dd]"
      

      QE Test

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini bucket_storage=couchstore,bucket_eviction_policy=fullEviction,rerun=False -t volumetests.Magma.volume.SystemTestMagma,nodes_init=3,replicas=1,skip_cleanup=True,num_items=100000000,num_buckets=1,bucket_names=GleamBook,doc_size=64,bucket_type=membase,compression_mode=off,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,maxttl=60,num_collections=20,doc_ops=create:update:delete:expiry,durability=None,pc=1,crashes=10,sdk_client_pool=True'
      

      Attachments

        For Gerrit Dashboard: MB-46090
        # Subject Branch Project Status CR V

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty