Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36356

[System Test] [Longevity] pausing of eventing handler is stuck since 15 hrs

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Test Blocker
    • Resolution: Duplicate
    • 6.5.0
    • 6.5.0
    • eventing
    • component cluster
    • Untriaged
    • Unknown

    Description

      Build : 6.5.0-4471
      Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Iteration : 1st

      Step:

      [2019-10-06T23:26:24-07:00, sequoiatools/eventing:6.5:cf2e4a] /eventing.py 172.23.98.135 8096 mad-hatter/bucket_op_n1ql.json Administrator password pause true
      warning using 'json' filter:  unexpected end of JSON input []
      warning using 'json' filter:  unexpected end of JSON input []
      

      Pausing eventing handler started at 2019-10-06T23:26:24-07:00 and below logs taken at 2019-10-07 01:11:08,585.

      logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.156.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.164.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.61.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.67.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.87.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.88.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.95.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.99.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.99.25.zip
      2019-10-07 01:11:08,585 - systestmon - INFO - ====== Log scan iteration number 39 complete. Sleeping for 3600 seconds ======

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          girish.benakappa Girish Benakappa created issue -
          girish.benakappa Girish Benakappa made changes -
          Field Original Value New Value
          Link This issue Clones MB-36341 [ MB-36341 ]
          girish.benakappa Girish Benakappa made changes -
          Assignee Deepkaran Salooja [ deepkaran.salooja ] Girish Benakappa [ girish.benakappa ]
          girish.benakappa Girish Benakappa made changes -
          Component/s eventing [ 14026 ]
          Component/s secondary-index [ 11211 ]
          girish.benakappa Girish Benakappa made changes -
          Description Build : 6.5.0-4471
          Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
          Scale : 3
          Iteration : 1st

          Step:
          {code:java}
          [2019-10-04T12:26:40-07:00, sequoiatools/couchbase-cli:6.5:94d558] failover -c 172.23.108.103:8091 --server-failover 172.23.106.100:8091 -u Administrator -p password --force
          [2019-10-04T12:27:26-07:00, sequoiatools/couchbase-cli:6.5:b16f29] rebalance -c 172.23.108.103:8091 -u Administrator -p password
          warning using 'json' filter: unexpected end of JSON input []


          Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]

          docker logs b16f29
          docker start b16f29

          *Unable to display progress bar on this os
          JERROR: Rebalance failed. See logs for detailed reason. You can try again.
          {code}


          panic: (diag.log in 172.23.104.69 and 172.23.104.87)
          {code:java}

          2019-10-04T12:32:10.581-07:00, ns_log:0:info:message(ns_1@172.23.104.69) - Service 'indexer' exited with status 2. Restarting. Messages:
          2019-10-04T12:32:09.130-07:00 [Info] default/default_claims/Backstore#17757113469883735399:0 Plasma: SMR reclaim pending is higher than expected: pending = 36 MB (expected = 10 MB), wCtxCnt = 30, changed reclaimList flush threshold from 12 to 13.
          panic: runtime error: index out of range

          goroutine 21707592 [running]:
          panic(0xf8fe20, 0xc4200160d0)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1
          github.com/couchbase/indexing/secondary/indexer.(*StreamState).updateRepairState(0xc42000fcc0, 0xc424430001, 0xc432425008, 0x8, 0xc4303a90f0, 0x3, 0x4, 0x0, 0x0, 0x0)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:528 +0x3a5
          github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendRestartMsg(0xc42001d280, 0x1b24da0, 0xc428205650)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3061 +0x1b61
          created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3003 +0xeba

          2019-10-04T12:32:10.766-07:00, ns_log:0:info:message(ns_1@172.23.104.87) - Service 'indexer' exited with status 2. Restarting. Messages:
          2019-10-04T12:32:10.611-07:00 [Info] ServiceMgr::cleanupRebalanceRunning Cleanup
          2019-10-04T12:32:10.611-07:00 [Info] ClustMgr:handleDelLocalValue Key RebalanceRunning
          2019-10-04T12:32:10.612-07:00 [Info] Rebalancer::observeRebalance exiting err <nil>
          panic: runtime error: index out of range

          goroutine 34975387 [running]:
          panic(0xf8fe20, 0xc4200160d0)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1
          github.com/couchbase/indexing/secondary/indexer.(*StreamState).updateRepairState(0xc423808000, 0xc425660001, 0xc435022118, 0x8, 0xc4324b21f0, 0x3, 0x4, 0x0, 0x0, 0x0)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:528 +0x3a5
          github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendRestartMsg(0xc420140c00, 0x1b24da0, 0xc433ef1ea0)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3061 +0x1b61
          created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3003 +0xeba

          -------------------------------


          {code}

          logs:

          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.156.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.157.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.164.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.61.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.69.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.70.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.87.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.106.188.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.108.103.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.148.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.251.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.252.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.253.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.56.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.95.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.119.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.121.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.122.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.239.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.242.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.98.135.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.11.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.20.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.21.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.25.zip
          Build : 6.5.0-4471
          Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
          Scale : 3
          Iteration : 1st

          Step:
          {code:java}
          [2019-10-06T23:26:24-07:00, sequoiatools/eventing:6.5:cf2e4a] /eventing.py 172.23.98.135 8096 mad-hatter/bucket_op_n1ql.json Administrator password pause true
          warning using 'json' filter: unexpected end of JSON input []
          warning using 'json' filter: unexpected end of JSON input []
          {code}

          Pausing eventing handler started at 2019-10-06T23:26:24-07:00 and below logs taken at 2019-10-07 01:11:08,585.

          logs:

          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.156.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.157.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.164.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.5.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.61.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.67.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.69.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.70.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.87.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.88.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.106.100.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.106.188.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.108.103.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.148.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.251.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.252.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.253.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.95.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.119.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.121.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.122.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.239.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.242.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.98.135.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.99.11.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.99.21.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.99.25.zip
          2019-10-07 01:11:08,585 - systestmon - INFO - ====== Log scan iteration number 39 complete. Sleeping for 3600 seconds ======
          girish.benakappa Girish Benakappa made changes -
          Link This issue Clones MB-36341 [ MB-36341 ]
          girish.benakappa Girish Benakappa made changes -
          Assignee Girish Benakappa [ girish.benakappa ] Jeelan Poola [ jeelan.poola ]
          jeelan.poola Jeelan Poola made changes -
          Assignee Jeelan Poola [ jeelan.poola ] Gautham Banasandra [ gautham.banasandra ]
          vikas.chaudhary Vikas Chaudhary made changes -
          Priority Critical [ 2 ] Test Blocker [ 6 ]
          jeelan.poola Jeelan Poola made changes -
          Resolution Duplicate [ 3 ]
          Status Open [ 1 ] Resolved [ 5 ]
          raju Raju Suravarjjala made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            Gautham.Banasandra Gautham Banasandra (Inactive)
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty