Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36340

[System Test] Rebalance exited with reason {{badmatch

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Build : 6.5.0-4471
      Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Iteration : 1st

      Step:

      [2019-10-04T10:18:51-07:00, sequoiatools/couchbase-cli:6.5:e082b3] failover -c 172.23.108.103:8091 --server-failover 172.23.104.67:8091 -u Administrator -p password --force
      [2019-10-04T10:19:05-07:00, sequoiatools/couchbase-cli:6.5:5bbe14] rebalance -c 172.23.108.103:8091 -u Administrator -p password
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
       
      docker logs 5bbe14
      docker start 5bbe14
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      

      Error: (diag.log in 172.23.108.103)

      2019-10-04T10:19:06.159-07:00, ns_orchestrator:0:info:message(ns_1@172.23.108.103) - Starting rebalance, KeepNodes = ['ns_1@172.23.104.156','ns_1@172.23.104.157',
                                       'ns_1@172.23.104.164','ns_1@172.23.104.61',
                                       'ns_1@172.23.104.69','ns_1@172.23.104.70',
                                       'ns_1@172.23.104.87','ns_1@172.23.106.100',
                                       'ns_1@172.23.106.188','ns_1@172.23.108.103',
                                       'ns_1@172.23.96.148','ns_1@172.23.96.251',
                                       'ns_1@172.23.96.252','ns_1@172.23.96.253',
                                       'ns_1@172.23.96.56','ns_1@172.23.96.95',
                                       'ns_1@172.23.97.119','ns_1@172.23.97.121',
                                       'ns_1@172.23.97.122','ns_1@172.23.97.239',
                                       'ns_1@172.23.97.242','ns_1@172.23.98.135',
                                       'ns_1@172.23.99.20','ns_1@172.23.99.21',
                                       'ns_1@172.23.99.25'], EjectNodes = [], Failed over and being ejected nodes = ['ns_1@172.23.104.67']; no delta recovery nodes; Operation Id = def4081507c372eebd57c71153ed3478
      2019-10-04T10:19:06.582-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.108.103) - Rebalance exited with reason {{badmatch,
                                        {error,
                                            {failed_nodes,['ns_1@172.23.97.119']}}},
                                    [{ns_janitor,cleanup_apply_config_body,4,
                                         [{file,"src/ns_janitor.erl"},{line,297}]},
                                     {ns_janitor,'-cleanup_apply_config/4-fun-0-',
                                         4,
                                         [{file,"src/ns_janitor.erl"},{line,217}]},
                                     {async,'-async_init/4-fun-1-',3,
                                         [{file,"src/async.erl"},{line,197}]}]}.
      Rebalance Operation Id = def4081507c372eebd57c71153ed3478
      
      

      logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.104.156.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.104.164.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.104.61.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.104.67.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.104.87.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.96.56.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.96.95.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.99.20.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.99.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570213139/collectinfo-2019-10-04T181901-ns_1%40172.23.99.25.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Some of the logs are missing on node 172.23.97.119. It appears that something deleted them while cbcollect_info was still running. Similarly, some of the vbuckets were deleted during the same time frame:

          Couchstore local documents (default, 713.couch.109)
          couch_dbdump --local /data/default/713.couch.109
          ==============================================================================
          Failed to open "/data/default/713.couch.109": no such file
          

          Due to this, it's impossible to investigate.

          Aliaksey Artamonau Aliaksey Artamonau (Inactive) added a comment - Some of the logs are missing on node 172.23.97.119. It appears that something deleted them while cbcollect_info was still running. Similarly, some of the vbuckets were deleted during the same time frame: Couchstore local documents (default, 713.couch.109) couch_dbdump --local /data/default/713.couch.109 ============================================================================== Failed to open "/data/default/713.couch.109": no such file Due to this, it's impossible to investigate.

          Bulk closing all invalid, duplicate and won't fix bugs. Please feel free to reopen them

          raju Raju Suravarjjala added a comment - Bulk closing all invalid, duplicate and won't fix bugs. Please feel free to reopen them

          People

            Aliaksey Artamonau Aliaksey Artamonau (Inactive)
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty