Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43484

Exception occurred in runloop during packet execution. Closing connection: GSL: Precondition failure at ../kv_engine/engines/ep/src/vbucket.cc:3229

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Steps:
      1. Create a 4 node cluster and magma bucket with replica=1

      +----------------+----------+--------------+
      		 
      | Nodes          | Services | Status       |
      		 
      +----------------+----------+--------------+
      		 
      | 172.23.106.218 | kv       | Cluster node |
      		 
      | 172.23.106.219 | ['kv']   | <--- IN ---  |
      		 
      | 172.23.106.95  | ['kv']   | <--- IN ---  |
      		 
      | 172.23.106.99  | ['kv']   | <--- IN ---  |
      		 
      +----------------+----------+--------------+

      2. Create a update:delete:expiry load with durability=PERSIST_TO_MAJORITY for a long duration.
      3. While load is running in step 2, on all 4 nodes keep lilling memcached 5 times with a wait of 3s in-between each kill.
      4. Wait for the bucket to warmup.
      5. Repeat Step 3, 4 again. This time the error messages is observed in the logs.

      172.23.106.218: 

      Found 'exception occurred in runloop' logs - ['2021-01-06T00:43:41.923510-08:00 WARNING 6872: exception occurred in runloop during packet execution. Closing connection: GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:3229. Cookies: [{"aiostat":"success","connection":"[ {\\"ip\\":\\"127.0.0.1\\",\\"port\\":32802} - {\\"ip\\":\\"127.0.0.1\\",\\"port\\":11209} (<ud>@ns_server</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":279,"cas":1609922495994134528,"datatype":["JSON"],"extlen":31,"key":"<ud>.test_docs-1608749</ud>","keylen":18,"magic":"ClientRequest","opaque":39,"opcode":"DCP_PREPARE","vbucket":804},"refcount":1}]\n']

      QE Test:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.32045.ini bucket_storage=magma,rerun=false,GROUP=P0;graceful_durability,randomize_value=true,doc_size=256,bucket_eviction_policy=fullEviction,nodes_init=4,graceful=True,upgrade_version=7.0.0-4122 -t magma.magma_crash_recovery.MagmaCrashTests.test_crash_during_recovery,num_items=5000000,doc_size=1024,graceful=True,sdk_timeout=60,doc_ops=update:delete:expiry,replicas=1,durability=PERSIST_TO_MAJORITY,GROUP=P0;graceful_durability'

      Logs attached!

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-46311 Magma sending stale items in backfill snapshot (GSL/exception thrown in VBucket::processSet with durability)

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty