Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7230

[system test] warmup access log corrupt during warmup in windows

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0, 2.2.0
    • Fix Version/s: 2.5.0
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
    • Environment:
      windows 2008 R2 64bit build 2.0.0-1952

      Description

      Environment:
      8 windows server 2008 R2 64 bit in ec2. Each server has 7.5 GB RAM and 45GB EBS disk to store data.
      Create a 6 nodes windows cluster in ec2

      Loading phase:

      • Create one default bucket and load 24 million items to it. Each item has size from 128 to 512 bytes.

      Access phase:

      • Mutate items with ratio creates/updates/gets/deletes/expirations = 10/60/20/5/5
      • Doing rebalance add node, remove node, failover, failover and add back.
      • The last test was reboot all 6 nodes in reboot warmup test. During warmup, I see error when collect raw warmup stats
        ep_warmup_access_log: corrupt

      root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.96.107.176:11210 raw warmup
      ep_warmup: enabled
      ep_warmup_dups: 0
      ep_warmup_estimate_time: 20704
      ep_warmup_estimated_key_count: 14538479
      ep_warmup_estimated_value_count: 14538479
      ep_warmup_key_count: 14538479
      ep_warmup_keys_time: 745292809
      ep_warmup_min_item_threshold: 100
      ep_warmup_min_memory_threshold: 100
      ep_warmup_oom: 0
      ep_warmup_state: done
      ep_warmup_thread: complete
      ep_warmup_time: 941489106
      ep_warmup_value_count: 2285132
      root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.110.222.56:11210 raw warmup
      ep_warmup: enabled
      ep_warmup_access_log: corrupt
      ep_warmup_dups: 0
      ep_warmup_estimate_time: 22194
      ep_warmup_estimated_key_count: 23969171
      ep_warmup_estimated_value_count: 23969171
      ep_warmup_key_count: 23969171
      ep_warmup_keys_time: 2466454551
      ep_warmup_min_item_threshold: 100
      ep_warmup_min_memory_threshold: 100
      ep_warmup_oom: 0
      ep_warmup_state: done
      ep_warmup_thread: complete
      ep_warmup_time: 2676578146
      ep_warmup_value_count: 976671
      root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.214.194.95:11210 raw warmup
      ep_warmup: enabled
      ep_warmup_dups: 0
      ep_warmup_estimate_time: 22131
      ep_warmup_estimated_key_count: 14540213
      ep_warmup_estimated_value_count: 14540213
      ep_warmup_key_count: 14540213
      ep_warmup_keys_time: 662356019
      ep_warmup_min_item_threshold: 100
      ep_warmup_min_memory_threshold: 100
      ep_warmup_oom: 0
      ep_warmup_state: done
      ep_warmup_thread: complete
      ep_warmup_time: 932353521
      ep_warmup_value_count: 2283472
      root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.46.181.58:11210 raw warmup
      Stats 'warmup' are not available from the requested engine.
      root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.110.218.124:11210 raw warmup
      Stats 'warmup' are not available from the requested engine.
      root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.111.66.215:11210 raw warmup
      ep_warmup: enabled
      ep_warmup_dups: 0
      ep_warmup_estimate_time: 21788
      ep_warmup_estimated_key_count: 14679821
      ep_warmup_estimated_value_count: 14679821
      ep_warmup_key_count: 14679821
      ep_warmup_keys_time: 638006011
      ep_warmup_min_item_threshold: 100
      ep_warmup_min_memory_threshold: 100
      ep_warmup_oom: 0
      ep_warmup_state: done
      ep_warmup_thread: complete
      ep_warmup_time: 785227909
      ep_warmup_value_count: 2264710
      root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.2.63.249:11210 raw warmup
      ep_warmup: enabled
      ep_warmup_access_log: corrupt
      ep_warmup_dups: 0
      ep_warmup_estimate_time: 26696
      ep_warmup_estimated_key_count: 24113568
      ep_warmup_estimated_value_count: 24113568
      ep_warmup_key_count: 24113568
      ep_warmup_keys_time: 1970642317
      ep_warmup_min_item_threshold: 100
      ep_warmup_min_memory_threshold: 100
      ep_warmup_oom: 0
      ep_warmup_state: done
      ep_warmup_thread: complete
      ep_warmup_time: 2221105121
      ep_warmup_value_count: 956974
      root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.110.206.19:11210 raw warmup
      ep_warmup: enabled
      ep_warmup_dups: 0
      ep_warmup_estimate_time: 20255
      ep_warmup_estimated_key_count: 24110614
      ep_warmup_estimated_value_count: unknown
      ep_warmup_key_count: 21603099
      ep_warmup_min_item_threshold: 100
      ep_warmup_min_memory_threshold: 100
      ep_warmup_oom: 0
      ep_warmup_state: loading keys
      ep_warmup_thread: running
      ep_warmup_value_count: 0

      Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1952-rel.setup.exe.manifest.xml

      # Subject Project Status CR V
      For Gerrit Dashboard: &For+MB-7230=message:MB-7230

        Activity

        Hide
        maria Maria McDuff (Inactive) added a comment -

        Have not been observed in latest 2.5.0 windows kv run.

        Show
        maria Maria McDuff (Inactive) added a comment - Have not been observed in latest 2.5.0 windows kv run.
        Hide
        thuan Thuan Nguyen added a comment -

        cluster has to be going to dgm (~60%) to be able so see bug.

        Show
        thuan Thuan Nguyen added a comment - cluster has to be going to dgm (~60%) to be able so see bug.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        re-opening to observe behavior in DGM of 60%

        Show
        maria Maria McDuff (Inactive) added a comment - re-opening to observe behavior in DGM of 60%
        Hide
        maria Maria McDuff (Inactive) added a comment -

        tony,

        pls verify with kv, kv+views system run.

        Show
        maria Maria McDuff (Inactive) added a comment - tony, pls verify with kv, kv+views system run.
        Hide
        thuan Thuan Nguyen added a comment -

        This bug is the same with bug MB-9864 and I re-assign it to Maria

        Show
        thuan Thuan Nguyen added a comment - This bug is the same with bug MB-9864 and I re-assign it to Maria

          People

          • Assignee:
            thuan Thuan Nguyen
            Reporter:
            thuan Thuan Nguyen
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes