Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47136

[Magma] : ERROR 428: Exception occurred during packet execution. Closing connection: PassiveDurabilityMonitor::completeSyncWrite vb:169 Pending resolution for 'SW @0x7f23c5492ff0

    XMLWordPrintable

Details

    Description

      Steps to Recreate:

      1. Create a 4 node cluster and magma bucket with replica=1
      2. Create a update:delete:expiry load with durability=MAJORITY_AND_PERSIST_TO_ACTIVE
      3. While load is running in step 2, on all 4 nodes keep klilling memcached 5 times with a wait of 3s in-between each kill.
      4. Wait for the bucket to warmup.
      5. Repeat Step 3, 4 again. This time the error messages is observed in the logs.

      Memcached Logs:
      Error found on Node 172.23.122.228

      ERROR 428: Exception occurred during packet execution. Closing connection: PassiveDurabilityMonitor::completeSyncWrite vb:169 Pending resolution for 'SW @0x7f23c5492ff0 qi:[key:'<ud>cid:0x0:test_docs-14317448</ud>' seqno:27225 reqs:{majority_and_persist_on_master, timeout=infinite}]  status:Pending', but received unexpected commit for key <ud>cid:0x0:test_docs-9004761</ud> and prepare seqno: 27247, highSeqno:27251, HPS:27250, HCS:27218, foundKeyWithSeqno:'SW @0x7f244c7a3c70 qi:[key:'<ud>cid:0x0:test_docs-9004761</ud>' seqno:27247 reqs:{majority_and_persist_on_master, timeout=infinite}]  status:Pending' lastReceivedSnapEnd:'none'. Cookies: [{"aiostat":"success","connection":"[ {\"ip\":\"127.0.0.1\",\"port\":53754} - {\"ip\":\"127.0.0.1\",\"port\":11209} (System, <ud>@ns_server</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":34,"cas":0,"datatype":"raw","extlen":16,"key":"<ud>.test_docs-9004761</ud>","keylen":18,"magic":"ClientRequest","opaque":86,"opcode":"DCP_COMMIT","vbucket":169},"refcount":1}]

      QE-TEST:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.63240.ini bucket_storage=magma,rerun=false,GROUP=P0;durability,randomize_value=true,doc_size=256,bucket_eviction_policy=fullEviction,nodes_init=4,graceful=False,enable_dp=True,collect_pcaps=True,get-cbcollect-info=True,infra_log_level=critical,log_level=error,bucket_storage=magma,upgrade_version=7.1.0-1038 -t storage.magma.magma_crash_recovery.MagmaCrashTests.test_crash_during_ops,num_items=10000000,doc_size=1024,sdk_timeout=60,graceful=False,doc_ops=create:update:delete:expiry,replicas=1,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,GROUP=P0;durability'

      Also observed core seen in MB-47106 along with above issue

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-47106 [Magma] - Non-negative counter exception in setBackfillRemaining_UNLOCKED

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-47140 PassiveDurabilityMonitor::completeSyncWrite vb:169 Pending resolution for 'SW @0x7f23c5492ff0 qi:[key:'<ud>cid:0x0:test_docs-14317448</ud>'

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ankush.sharma Ankush Sharma
              ankush.sharma Ankush Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty