Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7299

[system test] database compaction crashed during and after rebalance

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Blocker
    • 2.0
    • 2.0
    • couchbase-bucket
    • Security Level: Public
    • Windows 2008 R2 SP1 64bit in EC2 build 2.0.0-1967

    Description

      Environment:
      8 windows 2008 R2 SP1 64bit in ec2 (each server has 4 core cpu, 15GB RAM, 70GB for data and 170GB for view)

      1. 10.158.47.154
      2. 10.159.31.68
      3. 10.158.47.189
      4. 10.159.13.46
      5. 10.159.31.76
      6. 10.159.31.101
      1. 10.158.45.173
      2. 10.159.31.91

      Create a 6 nodes cluster.
      Create a default bucket and load 45+ million items with size from 128 to 512 bytes.
      Create one doc with 2 views. Let initial index complete.
      Mutate items with new size from 512 to 1024 bytes (creates/updates/gets/deletes/expirations = 10/60/20/5/5)
      Query view with rate 400 queries/second
      Add node 10.158.45.173 to cluster and rebalance. During rebalance, monitor data and view size.
      I saw data size in node 10.158.47.189 going up to big
      Change data compaction setting to 2% (force data compaction) but database size is still high.
      When rebalance done, database size of node 10.158.47.189 did not go down.
      I check diag log of node 10.158.47.189, I see a lot of crashed on database compaction process.

      =========================CRASH REPORT=========================
      crasher:
      initial call: compaction_daemon:spawn_vbucket_compactor/2-fun-0/0
      pid: <0.9691.327>
      registered_name: []
      exception error: no match of right hand side value none
      in function compaction_daemon:free_space/1
      in call from compaction_daemon:ensure_can_db_compact/1
      in call from compaction_daemon:'spawn_vbucket_compactor/2-fun-0'/4
      ancestors: [<0.9688.327>,<0.9685.327>,<0.9684.327>,compaction_daemon,
      <0.6769.208>,ns_server_sup,ns_server_cluster_sup,<0.67.0>]
      messages: []
      links: [<0.9688.327>]
      dictionary: []
      trap_exit: false
      status: running
      heap_size: 2584
      stack_size: 24
      reductions: 375
      neighbours:

      Link to mainifest file of build 2.0.0-1967 http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1967-rel.setup.exe.manifest.xml
      I will upload collect info soon

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty