Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-4363

dispatcher gives more priority to vbucket deletion even though disk write queue has > 1M items

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.0, 1.7.1, 1.7.1.1
    • Fix Version/s: 1.7.2
    • Component/s: None
    • Security Level: Public
    • Labels:
      None

      Description

      known issue with dispatcher scheduler.

      during rebalance disk write queue size grows to more than 1M items and due to this bug the dispatcher on some nodes keeps doing the vbucket deletion on different vbuckets over and over again andthis causes issues and make rebalancing very slow

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        i have monitored this node for the last 40 minutes and disk write queue size remained 1013k and vbucket-deletion task has been running and deleted more than 20 vbucket.s

        Show
        farshid Farshid Ghods (Inactive) added a comment - i have monitored this node for the last 40 minutes and disk write queue size remained 1013k and vbucket-deletion task has been running and deleted more than 20 vbucket.s
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        stats :

        accepting_conns: 1
        auth_cmds: 1088
        auth_errors: 0
        bucket_active_conns: 1
        bucket_conns: 20
        bytes_read: 63442439125
        bytes_written: 6603510477
        cas_badval: 0
        cas_hits: 0
        cas_misses: 0
        cmd_flush: 0
        cmd_get: 7531306
        cmd_set: 26812344
        conn_yields: 429139
        connection_structures: 92
        curr_connections: 28
        curr_items: 308258
        curr_items_tot: 1285805
        daemon_connections: 10
        decr_hits: 0
        decr_misses: 0
        delete_hits: 0
        delete_misses: 0
        ep_bg_fetched: 165
        ep_bg_load: 879409
        ep_bg_load_avg: 5329
        ep_bg_max_load: 140610
        ep_bg_max_wait: 33345
        ep_bg_min_load: 40
        ep_bg_min_wait: 39
        ep_bg_num_samples: 165
        ep_bg_wait: 81150
        ep_bg_wait_avg: 491
        ep_commit_num: 61737
        ep_commit_time: 0
        ep_commit_time_total: 24086
        ep_data_age: 1
        ep_data_age_highwat: 2144
        ep_db_cleaner_status: complete
        ep_db_strategy: multiMTVBDB
        ep_dbinit: 0
        ep_dbname: /mnt/default-data/default
        ep_dbshards: 4
        ep_diskqueue_drain: 24648
        ep_diskqueue_fill: 1058203
        ep_diskqueue_items: 1033566
        ep_diskqueue_memory: 90953808
        ep_diskqueue_pending: 1043187366
        ep_expired: 0
        ep_flush_all: false
        ep_flush_duration: 0
        ep_flush_duration_highwat: 775
        ep_flush_duration_total: 27480
        ep_flush_preempts: 0
        ep_flusher_state: running
        ep_flusher_todo: 0
        ep_io_num_read: 215273
        ep_io_num_write: 54748601
        ep_io_read_bytes: 222955770
        ep_io_write_bytes: 50648016228
        ep_item_begin_failed: 0
        ep_item_commit_failed: 0
        ep_item_flush_expired: 0
        ep_item_flush_failed: 0
        ep_items_rm_from_checkpoints: 49088066
        ep_kv_size: 1640026847
        ep_latency_arith_cmd: 0
        ep_latency_get_cmd: 7532472
        ep_latency_store_cmd: 26812344
        ep_max_data_size: 34902900736
        ep_max_txn_size: 1000
        ep_mem_high_wat: 26177175552
        ep_mem_low_wat: 20941740441
        ep_min_data_age: 0
        ep_num_active_non_resident: 0
        ep_num_checkpoint_remover_runs: 10016
        ep_num_eject_failures: 22978
        ep_num_eject_replicas: 6131491
        ep_num_expiry_pager_runs: 13
        ep_num_non_resident: 18008
        ep_num_not_my_vbuckets: 6263
        ep_num_pager_runs: 0
        ep_num_value_ejects: 6134897
        ep_onlineupdate: false
        ep_onlineupdate_revert_add: 0
        ep_onlineupdate_revert_delete: 0
        ep_onlineupdate_revert_update: 0
        ep_oom_errors: 0
        ep_overhead: 127924533
        ep_pending_ops: 0
        ep_pending_ops_max: 2
        ep_pending_ops_max_duration: 163767
        ep_pending_ops_total: 4
        ep_queue_age_cap: 900
        ep_queue_size: 1033566
        ep_storage_age: 1
        ep_storage_age_highwat: 2144
        ep_storage_type: featured
        ep_store_max_concurrency: 10
        ep_store_max_readers: 9
        ep_store_max_readwrite: 1
        ep_tap_bg_fetch_requeued: 0
        ep_tap_bg_fetched: 214608
        ep_tap_keepalive: 300
        ep_tmp_oom_errors: 0
        ep_too_old: 627317
        ep_too_young: 0
        ep_total_cache_size: 1393281051
        ep_total_del_items: 0
        ep_total_enqueued: 54832410
        ep_total_new_items: 26434634
        ep_total_persisted: 54748601
        ep_uncommitted_items: 0
        ep_value_size: 1458047980
        ep_vb_total: 22
        ep_vbucket_del: 158
        ep_vbucket_del_avg_walltime: 69102409
        ep_vbucket_del_fail: 0
        ep_vbucket_del_max_walltime: 134101378
        ep_vbucket_del_total_walltime: 10918180750
        ep_version: 1.7.0_56_g8d17d20
        ep_warmed_up: 0
        ep_warmup: true
        ep_warmup_dups: 0
        ep_warmup_oom: 0
        ep_warmup_thread: complete
        ep_warmup_time: 13369
        get_hits: 5784337
        get_misses: 1746969
        incr_hits: 0
        incr_misses: 0
        libevent: 2.0.11-stable
        limit_maxbytes: 67108864
        listen_disabled_num: 0
        mem_used: 1767951380
        pid: 6866
        pointer_size: 64
        rejected_conns: 0
        rusage_system: 3084.488768
        rusage_user: 2890.868668
        tap_checkpoint_end_received: 5657
        tap_checkpoint_end_sent: 5766
        tap_checkpoint_start_received: 5785
        tap_checkpoint_start_sent: 5996
        tap_connect_received: 248
        tap_mutation_received: 38494383
        tap_mutation_sent: 41704562
        tap_opaque_received: 408
        tap_opaque_sent: 653
        tap_vbucket_set_received: 96
        tap_vbucket_set_sent: 224
        threads: 4
        time: 1318884291
        total_connections: 1982
        uptime: 51910
        vb_active_curr_items: 308258
        vb_active_eject: 0
        vb_active_ht_memory: 3146688
        vb_active_itm_memory: 322166374
        vb_active_num: 2
        vb_active_num_non_resident: 0
        vb_active_ops_create: 0
        vb_active_ops_delete: 0
        vb_active_ops_reject: 0
        vb_active_ops_update: 0
        vb_active_perc_mem_resident: 100
        vb_active_queue_age: 1648144677000
        vb_active_queue_drain: 0
        vb_active_queue_fill: 342206
        vb_active_queue_memory: 30114128
        vb_active_queue_pending: 349263252
        vb_active_queue_size: 342206
        vb_dead_num: 2
        vb_pending_curr_items: 545398
        vb_pending_eject: 0
        vb_pending_ht_memory: 4327856
        vb_pending_itm_memory: 545353810
        vb_pending_num: 6
        vb_pending_num_non_resident: 0
        vb_pending_ops_create: 0
        vb_pending_ops_delete: 0
        vb_pending_ops_reject: 0
        vb_pending_ops_update: 0
        vb_pending_perc_mem_resident: 100
        vb_pending_queue_age: 2763075692000
        vb_pending_queue_drain: 0
        vb_pending_queue_fill: 545398
        vb_pending_queue_memory: 47995024
        vb_pending_queue_pending: 549576622
        vb_pending_queue_size: 545398
        vb_replica_curr_items: 432149
        vb_replica_eject: 102850
        vb_replica_ht_memory: 2586144
        vb_replica_itm_memory: 430624163
        vb_replica_num: 12
        vb_replica_num_non_resident: 18008
        vb_replica_ops_create: 1128
        vb_replica_ops_delete: 0
        vb_replica_ops_reject: 0
        vb_replica_ops_update: 23520
        vb_replica_perc_mem_resident: 95
        vb_replica_queue_age: 769715487000
        vb_replica_queue_drain: 24648
        vb_replica_queue_fill: 170599
        vb_replica_queue_memory: 12844656
        vb_replica_queue_pending: 144347492
        vb_replica_queue_size: 145962
        version: 1.4.4_461_gf99c147

        Show
        farshid Farshid Ghods (Inactive) added a comment - stats : accepting_conns: 1 auth_cmds: 1088 auth_errors: 0 bucket_active_conns: 1 bucket_conns: 20 bytes_read: 63442439125 bytes_written: 6603510477 cas_badval: 0 cas_hits: 0 cas_misses: 0 cmd_flush: 0 cmd_get: 7531306 cmd_set: 26812344 conn_yields: 429139 connection_structures: 92 curr_connections: 28 curr_items: 308258 curr_items_tot: 1285805 daemon_connections: 10 decr_hits: 0 decr_misses: 0 delete_hits: 0 delete_misses: 0 ep_bg_fetched: 165 ep_bg_load: 879409 ep_bg_load_avg: 5329 ep_bg_max_load: 140610 ep_bg_max_wait: 33345 ep_bg_min_load: 40 ep_bg_min_wait: 39 ep_bg_num_samples: 165 ep_bg_wait: 81150 ep_bg_wait_avg: 491 ep_commit_num: 61737 ep_commit_time: 0 ep_commit_time_total: 24086 ep_data_age: 1 ep_data_age_highwat: 2144 ep_db_cleaner_status: complete ep_db_strategy: multiMTVBDB ep_dbinit: 0 ep_dbname: /mnt/default-data/default ep_dbshards: 4 ep_diskqueue_drain: 24648 ep_diskqueue_fill: 1058203 ep_diskqueue_items: 1033566 ep_diskqueue_memory: 90953808 ep_diskqueue_pending: 1043187366 ep_expired: 0 ep_flush_all: false ep_flush_duration: 0 ep_flush_duration_highwat: 775 ep_flush_duration_total: 27480 ep_flush_preempts: 0 ep_flusher_state: running ep_flusher_todo: 0 ep_io_num_read: 215273 ep_io_num_write: 54748601 ep_io_read_bytes: 222955770 ep_io_write_bytes: 50648016228 ep_item_begin_failed: 0 ep_item_commit_failed: 0 ep_item_flush_expired: 0 ep_item_flush_failed: 0 ep_items_rm_from_checkpoints: 49088066 ep_kv_size: 1640026847 ep_latency_arith_cmd: 0 ep_latency_get_cmd: 7532472 ep_latency_store_cmd: 26812344 ep_max_data_size: 34902900736 ep_max_txn_size: 1000 ep_mem_high_wat: 26177175552 ep_mem_low_wat: 20941740441 ep_min_data_age: 0 ep_num_active_non_resident: 0 ep_num_checkpoint_remover_runs: 10016 ep_num_eject_failures: 22978 ep_num_eject_replicas: 6131491 ep_num_expiry_pager_runs: 13 ep_num_non_resident: 18008 ep_num_not_my_vbuckets: 6263 ep_num_pager_runs: 0 ep_num_value_ejects: 6134897 ep_onlineupdate: false ep_onlineupdate_revert_add: 0 ep_onlineupdate_revert_delete: 0 ep_onlineupdate_revert_update: 0 ep_oom_errors: 0 ep_overhead: 127924533 ep_pending_ops: 0 ep_pending_ops_max: 2 ep_pending_ops_max_duration: 163767 ep_pending_ops_total: 4 ep_queue_age_cap: 900 ep_queue_size: 1033566 ep_storage_age: 1 ep_storage_age_highwat: 2144 ep_storage_type: featured ep_store_max_concurrency: 10 ep_store_max_readers: 9 ep_store_max_readwrite: 1 ep_tap_bg_fetch_requeued: 0 ep_tap_bg_fetched: 214608 ep_tap_keepalive: 300 ep_tmp_oom_errors: 0 ep_too_old: 627317 ep_too_young: 0 ep_total_cache_size: 1393281051 ep_total_del_items: 0 ep_total_enqueued: 54832410 ep_total_new_items: 26434634 ep_total_persisted: 54748601 ep_uncommitted_items: 0 ep_value_size: 1458047980 ep_vb_total: 22 ep_vbucket_del: 158 ep_vbucket_del_avg_walltime: 69102409 ep_vbucket_del_fail: 0 ep_vbucket_del_max_walltime: 134101378 ep_vbucket_del_total_walltime: 10918180750 ep_version: 1.7.0_56_g8d17d20 ep_warmed_up: 0 ep_warmup: true ep_warmup_dups: 0 ep_warmup_oom: 0 ep_warmup_thread: complete ep_warmup_time: 13369 get_hits: 5784337 get_misses: 1746969 incr_hits: 0 incr_misses: 0 libevent: 2.0.11-stable limit_maxbytes: 67108864 listen_disabled_num: 0 mem_used: 1767951380 pid: 6866 pointer_size: 64 rejected_conns: 0 rusage_system: 3084.488768 rusage_user: 2890.868668 tap_checkpoint_end_received: 5657 tap_checkpoint_end_sent: 5766 tap_checkpoint_start_received: 5785 tap_checkpoint_start_sent: 5996 tap_connect_received: 248 tap_mutation_received: 38494383 tap_mutation_sent: 41704562 tap_opaque_received: 408 tap_opaque_sent: 653 tap_vbucket_set_received: 96 tap_vbucket_set_sent: 224 threads: 4 time: 1318884291 total_connections: 1982 uptime: 51910 vb_active_curr_items: 308258 vb_active_eject: 0 vb_active_ht_memory: 3146688 vb_active_itm_memory: 322166374 vb_active_num: 2 vb_active_num_non_resident: 0 vb_active_ops_create: 0 vb_active_ops_delete: 0 vb_active_ops_reject: 0 vb_active_ops_update: 0 vb_active_perc_mem_resident: 100 vb_active_queue_age: 1648144677000 vb_active_queue_drain: 0 vb_active_queue_fill: 342206 vb_active_queue_memory: 30114128 vb_active_queue_pending: 349263252 vb_active_queue_size: 342206 vb_dead_num: 2 vb_pending_curr_items: 545398 vb_pending_eject: 0 vb_pending_ht_memory: 4327856 vb_pending_itm_memory: 545353810 vb_pending_num: 6 vb_pending_num_non_resident: 0 vb_pending_ops_create: 0 vb_pending_ops_delete: 0 vb_pending_ops_reject: 0 vb_pending_ops_update: 0 vb_pending_perc_mem_resident: 100 vb_pending_queue_age: 2763075692000 vb_pending_queue_drain: 0 vb_pending_queue_fill: 545398 vb_pending_queue_memory: 47995024 vb_pending_queue_pending: 549576622 vb_pending_queue_size: 545398 vb_replica_curr_items: 432149 vb_replica_eject: 102850 vb_replica_ht_memory: 2586144 vb_replica_itm_memory: 430624163 vb_replica_num: 12 vb_replica_num_non_resident: 18008 vb_replica_ops_create: 1128 vb_replica_ops_delete: 0 vb_replica_ops_reject: 0 vb_replica_ops_update: 23520 vb_replica_perc_mem_resident: 95 vb_replica_queue_age: 769715487000 vb_replica_queue_drain: 24648 vb_replica_queue_fill: 170599 vb_replica_queue_memory: 12844656 vb_replica_queue_pending: 144347492 vb_replica_queue_size: 145962 version: 1.4.4_461_gf99c147
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        its been an hour and half now and dispatcher is still removing items from vbucket

        Show
        farshid Farshid Ghods (Inactive) added a comment - its been an hour and half now and dispatcher is still removing items from vbucket
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        fixed in 1.7.2

        Show
        farshid Farshid Ghods (Inactive) added a comment - fixed in 1.7.2

          People

          • Assignee:
            chiyoung Chiyoung Seo
            Reporter:
            farshid Farshid Ghods (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes