Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46221

memcached crash in CouchKVStore::commit

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown
    • KV-Engine CC Final Sprint

    Description

      In the 4B rebalance runs, we saw auto failover happened during the load phase. We increased failover threshold from 5 seconds to 30 seconds in the second run (rebalance-in), but the run still hit auto failover. Both runs didn't hit the issue with 6.6.2 build.

       

      Rebalance-out (min), 5 -> 4, 4B x 1KB, 15K ops/sec (90/10 R/W), 10%% cache miss rate

      Build: 7.0.0-5071

      Job: http://perf.jenkins.couchbase.com/job/titan-reb/2119/ 

      [user:info,2021-05-05T12:00:57.796-07:00,ns_1@172.23.96.100:<0.5861.0>:auto_failover:log_failover_success:544]Node ('ns_1@172.23.96.100') was automatically failed over. Reason: The data service did not respond for the duration of the auto-failover threshold. Either none of the buckets have warmed up or there is an issue with the data service.

       

      Rebalance-in (min), 4 -> 5, 4B x 1KB, 15K ops/sec (90/10 R/W), 10%% cache miss rate

      Build: 7.0.0-5071

      Job: http://perf.jenkins.couchbase.com/job/titan-reb/2124/ 

      [user:info,2021-05-07T20:39:41.757-07:00,ns_1@172.23.96.100:<0.6131.0>:auto_failover:log_failover_success:544]Node ('ns_1@172.23.96.102') was automatically failed over. Reason: The data service did not respond for the duration of the auto-failover threshold. Either none of the buckets have warmed up or there is an issue with the data service.

      Attachments

        1. 101.bt_txt
          47 kB
        2. 103.txt
          3 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              bo-chun.wang Bo-Chun Wang
              bo-chun.wang Bo-Chun Wang
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty