Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44366

[System Test] Rebalance failure observed with indexer panic in longevity

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.0.0-4454

      Test:
      -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      1st iteration

      From .97.74 diag.log:

      ns_vbucket_mover:0:info:message(ns_1@172.23.97.74) - Bucket "default" rebalance appears to be swap rebalance
      2021-02-14T13:38:51.554-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.97.74) - Rebalance exited with reason {service_rebalance_failed,index,
                                       {agent_died,<30392.30806.0>,
                                           {lost_connection,shutdown}}}.
      Rebalance Operation Id = e10cdb9eef30f4793dc9bb40e785d795
      2021-02-14T13:38:51.569-08:00, ns_log:0:info:message(ns_1@172.23.96.243) - Service 'indexer' exited with status 2. Restarting. Messages:
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insertSecIndex(0xc01766cc00, 0xc019c1cca0, 0xd, 0x10, 0xc0080ad3a0, 0x12, 0x20, 0x7, 0x4c2400, 0xc00b798800, ...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:775 +0xb18
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insert(0xc01766cc00, 0xc019c1cca0, 0xd, 0x10, 0xc0080ad3a0, 0x12, 0x20, 0x7, 0x400, 0xc00b798800, ...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:733 +0x164
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker(0xc01766cc00, 0x7)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:610 +0x58a
      created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initWriters
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2969 +0x521
       
      2021-02-14T13:39:24.214-08:00, indexing:0:info:message(ns_1@172.23.96.243) - Detected storage corruption for index default_rating, partition id 0. Starting cleanup.
      2021-02-14T13:39:24.411-08:00, indexing:0:info:message(ns_1@172.23.96.243) - Cleanup done for index default_rating, partition id 0.
      

      Logs:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.106.134.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.120.58.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.120.73.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.120.74.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.120.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.120.81.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.120.86.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.121.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.123.24.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.123.25.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.123.26.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.123.31.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.123.32.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.123.33.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.96.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.96.14.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.96.243.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.96.254.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.96.48.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.97.105.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.97.110.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.97.112.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.97.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.97.149.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.97.150.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.97.151.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.97.241.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613341716/collectinfo-2021-02-14T222838-ns_1%40172.23.97.74.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jeelan.poola Jeelan Poola
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty