Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-26550

Memcached crash during rebalance (was: Swap rebalance (7 nodes, 10B items) gets stuck)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 5.5.0
    • 5.5.0
    • memcached
    • E5-2680 v3 (48 vCPU)
      64 GB
      Samsung PM863a (SATA SSD)

    Description

      Test case:

      • 7 nodes
      • 1 bucket, 1 replica
      • 10B items (~10TB before compression)
      • 10K ops/sec (read-heavy), 10% cache miss ratio
      • Swap rebalance of one of the nodes (172.23.96.108 -> 172.23.96.109)

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_510-1323_rebalance_d902

      I tried to restart rebalance after the test but it failed:

      Rebalance exited with reason {{badmatch,
      		 
      {error,
      		 
      {failed_nodes,['ns_1@172.23.96.109']}}},
      		 
      [{ns_janitor,cleanup_with_states,6,
      		 
      [{file,"src/ns_janitor.erl"},{line,136}]},
      		 
      {ns_rebalancer,do_run_janitor_pre_rebalance,1,
      		 
      [{file,"src/ns_rebalancer.erl"},
      		 
      {line,766}]}]}

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-26633 memcached crash during initial data load (10TB)

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              dhaikney David Haikney (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty