Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-24208

[Ephemeral] Memcached fails and buckets are left in warmup state while loading data on Ephemeral buckets.

    XMLWordPrintable

Details

    Description

      Memcached exists abruptly and causes rebalance failures and causes the ephemeral buckets to go on warmup state when loading data on to the ephemeral buckets. 

      The ns_server tests are failing/getting struck due to this issue and blocking the test runs.

      In the test, we create ephemeral buckets and then load data with multiple clients connecting to the server. The log suggests that memcached exit due to high number of clients being connected. But memcached fails to come back and causes buckets to go on warmup state causing the whole test to fail. Attaching the logs of the cluster for investigation.

      http://qa.sc.couchbase.com/job/cen006-nserv-vset05-01-rebalanceinout_P1_set2-ephemeral/2/consoleFull is one of the test suite that is failing due to this issue. Note that this issue is very sporadic and running the same test might not result in the failure. But the issue is seen mostly when we load multiple ephemeral buckets with high number of connections. 

      Repro steps:

      1. Clone/update testrunner and create an ini file with the nodes ip.
      2. Run following test with the ini file create.
      3. ./testrunner -i <ini file> get-cbcollect-info=False,get-logs=False,stop-on-failure=False,get-coredumps=False,force_kill_memached=False,verify_unacked_bytes=True,std_vbucker_dist=8,chk_max_items=1000,chk_period=30,bucket_type=ephemeral -t rebalance.rebalanceinout.RebalanceInOutTests.test_incremental_rebalance_out_in_with_mutation,init_num_nodes=3,items=400000,standard_buckets=2,sasl_buckets=2,standard_bucket_priority=low:high,sasl_bucket_priority=low:high,zone=2

      Attachments

        Issue Links

          Activity

            People

              bharath.gp Bharath G P
              bharath.gp Bharath G P
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty