Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30832

[System Test]: Rebalance failed with config_sync_failed

    XMLWordPrintable

Details

    • Bug
    • Resolution: User Error
    • Critical
    • 6.0.0
    • 6.0.0
    • ns_server
    • None
    • centos longevity

    Description

      Build: 6.0.0-1458

      Longevity :

      1. Create 22 node cluster  (9 KV,5 index, 2 query , 2 fts, 2 eventing, 2 cbas)
      2. Create 10 buckets (default bucket with Active compression)
      3. Create views
      4. Load data
      5. Remove kv node
      6. Deploy eventing functions
      7. Create dataset on analytics on 4 buckets
      8. Create index on 2 datasets
      9. Create 2i index
      10. Load more data
      11. Run queries on 2i
      12. Swap a KV node
      13. Run 240 queries per second on analytics
      14. Connect link Local
      15. Load more data to default bucket
      16. Add eventing node
      17. Remove eventing node
      18. Swap eventing node
      19. Disconnect link Local 
      20. Add analytics node 
      21. connect link Local
      22. Disconnect link Local 
      23. Remove analytics node
      24. connect link Local
      25. Swap analytics node
      26. Kill analytics nodes
      27. Run views
      28. Create fts indexes 
      29. Regex search on FTS
      30. XDCR replication
      31. Add rbac users
      32. Undeploy eventing handlers
      33. Load 1M doc
      34. Create 2i indexes 
      35. Rebalance in index
      36. Rebalance out index
      37. Swap index node
      38. Rebalance in 2 index nodes
      39. Rebalance out 2 index nodes
      40. Rebalance out 1 KV
      41. Rebalance in 1 KV
      42. Failover -> Full recovery index node
      43. Failover -> Rebalance out index node
      44. Add index node
      45. Redeploy eventing handlers
      46. Run Tpcc
      47. Update Doc
      48. Add a kv node , failover kv node -> rebalance 
      49. swap hard failover -> Add 1 KV remove 2 KV as soft and hard failover
      50. Multinode autofailover -> failover 3 KV nodes and rebalance 

      [2018-08-09T01:17:44-07:00, sequoiatools/couchbase-cli:a5bbd0] setting-autofailover -c 172.23.108.103:8091 -u Administrator -p password --enable-auto-failover=1 --auto-failover-timeout=5 --max-failovers=3
      		 
      [2018-08-09T01:18:01-07:00, sequoiatools/cmd:8d8aac] 10
      		 
      [pull] sequoiatools/cbutil
      		 
      [2018-08-09T01:18:24-07:00, sequoiatools/cbutil:9b49cc] /cbinit.py 172.23.108.104 root couchbase stop
      		 
      [2018-08-09T01:19:15-07:00, sequoiatools/cmd:0f43e5] 10
      		 
      [2018-08-09T01:19:39-07:00, sequoiatools/cbutil:697137] /cbinit.py 172.23.96.148 root couchbase stop
      		 
      [2018-08-09T01:20:16-07:00, sequoiatools/cmd:705a85] 10
      		 
      [2018-08-09T01:20:38-07:00, sequoiatools/cbutil:146195] /cbinit.py 172.23.97.239 root couchbase stop
      		 
      [2018-08-09T01:21:05-07:00, sequoiatools/cmd:76a34f] 10
      		 
      [2018-08-09T01:21:27-07:00, sequoiatools/couchbase-cli:87b0d2] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      		 
      →  
       
      		 
      Error occurred on container - sequoiatools/couchbase-cli:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
      		 
       
      		 
      docker logs 87b0d2
      		 
      docker start 87b0d2
      		 
       
      		 
      *Unable to display progress bar on this os

      Nodes should be rebalance out but it failed with the following error

      Rebalance exited with reason {config_sync_failed, {error, {get_compressed_failed, [{'ns_1@172.23.97.239', {exit, {{nodedown,'ns_1@172.23.97.239'}, {gen_server,call, [{ns_config_remote,'ns_1@172.23.97.239'}, get_compressed,infinity]}}}}, {'ns_1@172.23.96.148', {exit, {{nodedown,'ns_1@172.23.96.148'}, {gen_server,call, [{ns_config_remote,'ns_1@172.23.96.148'}, get_compressed,infinity]}}}}]}}}ns_orchestrator 000ns_1@172.23.108.103
      		 
      1:21:29 AM   Thu Aug 9, 2018

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            vikas.chaudhary Vikas Chaudhary
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty