Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-1541

Missing data under load

    XMLWordPrintable

Details

    • Bug
    • Resolution: Incomplete
    • 1.6.0 beta2
    • 1.6.0 beta1
    • couchbase-bucket
    • None
    • Operating System: All
      Platform: All

    Description

      This seems quite basic load test on disk > ram, and quite easy to reproduce. Randi have you run such test so far??

      1. I installed the latest version: 1.6.0beta2rc1_50_g6d70697
      2. Run ./memcachetest -h 10.211.55.5 -i 10000000 (long load test)
      3. Watched the minute stats which increased the load.
      4. Ram got to the maximum on quota on this vm (~840M)
      5. Disk writes peaked at 361K (according to the graph).
      6. On the test program I start seeing a lot of the following error :
      missing data for <06226345>
      missing data for <03077567>
      missing data for <02659817>
      missing data for <03144154>
      missing data for <06829352>
      missing data for <09490985>
      missing data for <03021173>
      ...
      7. The VM became extremely slow
      8. Here is the top command:

      top - 23:37:15 up 7:11, 3 users, load average: 7.13, 5.63, 3.36
      Tasks: 127 total, 2 running, 123 sleeping, 0 stopped, 2 zombie
      Cpu(s): 1.3%us, 0.6%sy, 0.0%ni, 0.0%id, 96.8%wa, 1.3%hi, 0.0%si, 0.0%st
      Mem: 1026392k total, 1019444k used, 6948k free, 1408k buffers
      Swap: 2064344k total, 923380k used, 1140964k free, 95596k cached

      PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
      7548 northsca 15 0 1133m 765m 1004 S 1.7 76.4 1:59.36 memcached
      2557 root 15 0 217m 6288 1836 S 0.3 0.6 0:15.89 Xorg

      9. Attached is the log file.

      I don't expect misses on disk > RAM so starting this as Major P1, let me know if my understanding is incorrect (making rebalancing work is still higher priority though).

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            trond@northscale.com Trond Norbye [X] (Inactive)
            sharon.barr@northscale.com sharon.barr@northscale.com
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty