Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45594

[System Test][couchbase-bucket] Rebalance failures with error buckets_cleanup_failed

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown
    • KV-Engine 2021-March

    Description

      7.0.0-4910

      Test:
      -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale 3
      Iteration 1

      Two rebalance failures observed with this error:

      2021-04-11T10:26:42.904-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.108.103) - Rebalance exited with reason {buckets_cleanup_failed,
                                       ['ns_1@172.23.97.239','ns_1@172.23.97.242',
                                        'ns_1@172.23.97.119']}.
      Rebalance Operation Id = bdd600f5cddb7bf8fa180aaaac694f63
      

      2021-04-11T06:55:18.109-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.108.103) - Rebalance exited with reason {buckets_cleanup_failed,['ns_1@172.23.97.119']}.
      Rebalance Operation Id = 95124008717744bd960d9b7f508dec3e
      

      Logs:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.104.137.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.104.155.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.104.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.104.67.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.105.107.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.105.111.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.120.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.121.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.123.27.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.123.28.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.99.20.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.99.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618168029/collectinfo-2021-04-11T190711-ns_1%40172.23.99.25.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            There is one more occurrence of this issue with build 7.0.0-5017. But i do not see any core files in any of the related nodes again.

            Log files :
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.104.137.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.104.155.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.104.157.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.104.5.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.104.67.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.104.69.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.105.107.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.106.188.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.108.103.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.120.245.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.121.117.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.121.3.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.123.27.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.123.28.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.96.148.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.96.251.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.96.252.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.96.253.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.97.119.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.97.121.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.97.122.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.97.239.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.97.242.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.98.135.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.99.11.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.99.20.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.99.21.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.99.25.zip

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - There is one more occurrence of this issue with build 7.0.0-5017. But i do not see any core files in any of the related nodes again. Log files : url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.104.137.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.104.155.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.104.157.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.104.5.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.104.67.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.104.69.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.105.107.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.106.188.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.108.103.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.120.245.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.121.117.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.121.3.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.123.27.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.123.28.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.96.148.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.96.251.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.96.252.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.96.253.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.97.119.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.97.121.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.97.122.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.97.239.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.97.242.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.98.135.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.99.11.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.99.20.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.99.21.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619283863/collectinfo-2021-04-24T170427-ns_1%40172.23.99.25.zip
            hareen.kancharla Hareen Kancharla added a comment - - edited

            Arunkumar Senthilnathan, Mihir Kamdar, Meni Hillel: Jotting this note down to avoid some confusion on the different symptoms seen on the 4 issues reported on this ticket. 

            1) Arun's first issue (in the ticket description), Arun's Second issue (reported on April 22nd) and Mihir's second issue (reported on April 24th) are all because of the following reason:

                Issue is due to a timeout in the chronicle_compat:get_snapshot/2 API (and chronicle_agent:get_latest_snapshot). Aliakesi is investigating timeouts similar to this via this ticket - MB-44824.

            2) Mihir's 1st issue : There is a process coredump in /data/<bucket-name> (DVJH) path and ns_server doesn't have permissions to delete it.

            This can be resolved easily, by configuring the unix kernels with a specific path to dump the core. You could do something similar as described in the link below, to prevent the core being written to the current working directory when a process crash happens. 

            https://man7.org/linux/man-pages/man5/core.5.html

            hareen.kancharla Hareen Kancharla added a comment - - edited Arunkumar Senthilnathan , Mihir Kamdar , Meni Hillel : Jotting this note down to avoid some confusion on the different symptoms seen on the 4 issues reported on this ticket.  1) Arun's first issue (in the ticket description), Arun's Second issue (reported on April 22nd) and Mihir's second issue (reported on April 24th) are all because of the following reason:     Issue is due to a timeout in the chronicle_compat:get_snapshot/2 API (and chronicle_agent:get_latest_snapshot). Aliakesi is investigating timeouts similar to this via this ticket - MB-44824 . 2) Mihir's 1st issue : There is a process coredump in /data/<bucket-name> (DVJH) path and ns_server doesn't have permissions to delete it. This can be resolved easily, by configuring the unix kernels with a specific path to dump the core. You could do something similar as described in the link below, to prevent the core being written to the current working directory when a process crash happens.  https://man7.org/linux/man-pages/man5/core.5.html

            Dave Finlay Meni Hillel we saw one more occurrence of issue #1 as described by Hareen above in the longevity test. Since the timeout is being investigated in MB-44824, can we convert MB-44824 into a bug from a task, so that this failure can be addressed effectively.

            Error seen in the error.log of 172.23.108.103:

            [rebalance:error,2021-04-29T12:38:43.628-07:00,ns_1@172.23.108.103:<0.2337.364>:ns_rebalancer:maybe_cleanup_old_buckets:941]Failed to cleanup old buckets on node 'ns_1@172.23.97.242': {badrpc,
                                                                         {'EXIT',timeout}}
            [rebalance:error,2021-04-29T12:38:43.628-07:00,ns_1@172.23.108.103:<0.2337.364>:ns_rebalancer:maybe_cleanup_old_buckets:941]Failed to cleanup old buckets on node 'ns_1@172.23.97.119': {badrpc,
                                                                         {'EXIT',timeout}}
            [user:error,2021-04-29T12:38:43.630-07:00,ns_1@172.23.108.103:<0.22663.0>:ns_orchestrator:log_rebalance_completion:1405]Rebalance exited with reason {buckets_cleanup_failed,
                                             ['ns_1@172.23.97.242','ns_1@172.23.97.119']}.
            Rebalance Operation Id = 66f6ae0bbf1d893fefea43076b3329a8
            

            Logs :
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.104.137.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.104.155.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.104.157.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.104.5.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.104.67.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.104.70.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.106.100.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.106.188.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.108.103.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.120.245.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.121.117.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.121.3.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.123.27.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.123.28.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.96.148.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.96.251.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.96.252.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.96.253.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.97.119.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.97.121.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.97.122.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.97.239.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.97.242.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.98.135.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.99.20.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.99.21.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.99.25.zip

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Dave Finlay Meni Hillel we saw one more occurrence of issue #1 as described by Hareen above in the longevity test. Since the timeout is being investigated in MB-44824 , can we convert MB-44824 into a bug from a task, so that this failure can be addressed effectively. Error seen in the error.log of 172.23.108.103: [rebalance:error,2021-04-29T12:38:43.628-07:00,ns_1@172.23.108.103:<0.2337.364>:ns_rebalancer:maybe_cleanup_old_buckets:941]Failed to cleanup old buckets on node 'ns_1@172.23.97.242': {badrpc, {'EXIT',timeout}} [rebalance:error,2021-04-29T12:38:43.628-07:00,ns_1@172.23.108.103:<0.2337.364>:ns_rebalancer:maybe_cleanup_old_buckets:941]Failed to cleanup old buckets on node 'ns_1@172.23.97.119': {badrpc, {'EXIT',timeout}} [user:error,2021-04-29T12:38:43.630-07:00,ns_1@172.23.108.103:<0.22663.0>:ns_orchestrator:log_rebalance_completion:1405]Rebalance exited with reason {buckets_cleanup_failed, ['ns_1@172.23.97.242','ns_1@172.23.97.119']}. Rebalance Operation Id = 66f6ae0bbf1d893fefea43076b3329a8 Logs : url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.104.137.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.104.155.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.104.157.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.104.5.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.104.67.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.104.70.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.106.100.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.106.188.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.108.103.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.120.245.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.121.117.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.121.3.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.123.27.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.123.28.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.96.148.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.96.251.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.96.252.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.96.253.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.97.119.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.97.121.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.97.122.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.97.239.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.97.242.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.98.135.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.99.20.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.99.21.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619730491/collectinfo-2021-04-29T210814-ns_1%40172.23.99.25.zip
            dfinlay Dave Finlay added a comment -

            Almost certain dupe of MB-46099. Keeping MB-46099 around as it's the most recent and also because it's on the 130 node cluster.

            dfinlay Dave Finlay added a comment - Almost certain dupe of MB-46099 . Keeping MB-46099 around as it's the most recent and also because it's on the 130 node cluster.

            Closing dupe issues

            arunkumar Arunkumar Senthilnathan (Inactive) added a comment - Closing dupe issues

            People

              Aliaksey Artamonau Aliaksey Artamonau (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty