Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-20943

Delete Bucket Waiting for clients to disconnect

    XMLWordPrintable

Details

    Description

      Removed a node and could not add back to cluster. Memcached Logs show that the node is waiting for a bucket delete to finish. The task gets down to 8 clients waiting. So far it's been in state for 3 hours.

      2016-09-15T08:41:30.690285-07:00 NOTICE 257 The connected bucket is being deleted.. disconnecting
      2016-09-15T08:41:30.692749-07:00 NOTICE 262 The connected bucket is being deleted.. disconnecting
      2016-09-15T08:41:30.692715-07:00 NOTICE 253 Delete bucket [default]. Wait for clients to disconnect
      2016-09-15T08:41:30.692864-07:00 NOTICE 3930456088 Delete bucket [default]. Still waiting: 58 clients connected
      2016-09-15T08:41:31.694093-07:00 NOTICE 3930456088 Delete bucket [default]. Still waiting: 8 clients connected
      ...
      2016-09-15T11:21:03.780160-07:00 NOTICE 3930456088 Delete bucket [default]. Still waiting: 8 clients connected
      2016-09-15T11:21:04.780107-07:00 NOTICE 3930456088 Delete bucket [default]. Still waiting: 8 clients connected
      
      

      ns_server logs 1 minute into waiting for bucket delete

      [ns_server:error,2016-09-15T08:42:00.147-07:00,ns_1@172.23.106.213:<0.4113.982>:timeout_diag_logger:do_diag:118]Got timeout {slow_bucket_stop,{{single_bucket_kv_sup,"default"},
                                     <0.21160.970>,supervisor,
                                     [single_bucket_kv_sup]}}
      Processes snapshot is: 
       
      [ns_server:error,2016-09-15T08:42:00.147-07:00,ns_1@172.23.106.213:<0.4113.982>:timeout_diag_logger:do_diag:120]
      {<0.31114.971>,
       [{registered_name,[]},
        {status,waiting},
        {initial_call,{inet_tcp_dist,do_setup,6}},
        {backtrace,[<<"Program counter: 0x00007fd27515d330 (dist_util:con_loop/9 + 112)">>,
                    <<"CP: 0x0000000000000000 (invalid)">>,<<"arity = 0">>,<<>>,
                    <<"0x00007fd22f60d3b0 Return addr 0x0000000000893588 (<terminate process normally>)">>,
                    <<"y(0)     []">>,<<"y(1)     #Fun<inet_tcp_dist.getstat.1>">>,
                    <<"y(2)     #Fun<inet_tcp_dist.tick.1>">>,
                    <<"y(3)     {tick,7430,7436,1,1}">>,<<"y(4)     normal">>,
                    <<"y(5)     'ns_1@172.23.106.213'">>,
      
      

      node is currently online in this state: 172.23.106.213
      https://s3.amazonaws.com/scalability-mcafee/collectinfo-2016-09-15T134105-ns_1%40172.23.106.213.zip

      ~6 days into longevity test

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              owend Daniel Owen
              tommie Tommie McAfee (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty