Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36356

[System Test] [Longevity] pausing of eventing handler is stuck since 15 hrs

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Test Blocker
    • Resolution: Duplicate
    • 6.5.0
    • 6.5.0
    • eventing
    • component cluster
    • Untriaged
    • Unknown

    Description

      Build : 6.5.0-4471
      Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Iteration : 1st

      Step:

      [2019-10-06T23:26:24-07:00, sequoiatools/eventing:6.5:cf2e4a] /eventing.py 172.23.98.135 8096 mad-hatter/bucket_op_n1ql.json Administrator password pause true
      warning using 'json' filter:  unexpected end of JSON input []
      warning using 'json' filter:  unexpected end of JSON input []
      

      Pausing eventing handler started at 2019-10-06T23:26:24-07:00 and below logs taken at 2019-10-07 01:11:08,585.

      logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.156.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.164.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.61.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.67.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.87.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.88.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.95.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.99.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.99.25.zip
      2019-10-07 01:11:08,585 - systestmon - INFO - ====== Log scan iteration number 39 complete. Sleeping for 3600 seconds ======

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          girish.benakappa Girish Benakappa created issue -
          girish.benakappa Girish Benakappa made changes -
          Field Original Value New Value
          Link This issue Clones MB-36341 [ MB-36341 ]
          girish.benakappa Girish Benakappa made changes -
          Assignee Deepkaran Salooja [ deepkaran.salooja ] Girish Benakappa [ girish.benakappa ]
          girish.benakappa Girish Benakappa made changes -
          Component/s eventing [ 14026 ]
          Component/s secondary-index [ 11211 ]
          girish.benakappa Girish Benakappa made changes -
          Description Build : 6.5.0-4471
          Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
          Scale : 3
          Iteration : 1st

          Step:
          {code:java}
          [2019-10-04T12:26:40-07:00, sequoiatools/couchbase-cli:6.5:94d558] failover -c 172.23.108.103:8091 --server-failover 172.23.106.100:8091 -u Administrator -p password --force
          [2019-10-04T12:27:26-07:00, sequoiatools/couchbase-cli:6.5:b16f29] rebalance -c 172.23.108.103:8091 -u Administrator -p password
          warning using 'json' filter: unexpected end of JSON input []


          Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]

          docker logs b16f29
          docker start b16f29

          *Unable to display progress bar on this os
          JERROR: Rebalance failed. See logs for detailed reason. You can try again.
          {code}


          panic: (diag.log in 172.23.104.69 and 172.23.104.87)
          {code:java}

          2019-10-04T12:32:10.581-07:00, ns_log:0:info:message(ns_1@172.23.104.69) - Service 'indexer' exited with status 2. Restarting. Messages:
          2019-10-04T12:32:09.130-07:00 [Info] default/default_claims/Backstore#17757113469883735399:0 Plasma: SMR reclaim pending is higher than expected: pending = 36 MB (expected = 10 MB), wCtxCnt = 30, changed reclaimList flush threshold from 12 to 13.
          panic: runtime error: index out of range

          goroutine 21707592 [running]:
          panic(0xf8fe20, 0xc4200160d0)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1
          github.com/couchbase/indexing/secondary/indexer.(*StreamState).updateRepairState(0xc42000fcc0, 0xc424430001, 0xc432425008, 0x8, 0xc4303a90f0, 0x3, 0x4, 0x0, 0x0, 0x0)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:528 +0x3a5
          github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendRestartMsg(0xc42001d280, 0x1b24da0, 0xc428205650)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3061 +0x1b61
          created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3003 +0xeba

          2019-10-04T12:32:10.766-07:00, ns_log:0:info:message(ns_1@172.23.104.87) - Service 'indexer' exited with status 2. Restarting. Messages:
          2019-10-04T12:32:10.611-07:00 [Info] ServiceMgr::cleanupRebalanceRunning Cleanup
          2019-10-04T12:32:10.611-07:00 [Info] ClustMgr:handleDelLocalValue Key RebalanceRunning
          2019-10-04T12:32:10.612-07:00 [Info] Rebalancer::observeRebalance exiting err <nil>
          panic: runtime error: index out of range

          goroutine 34975387 [running]:
          panic(0xf8fe20, 0xc4200160d0)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1
          github.com/couchbase/indexing/secondary/indexer.(*StreamState).updateRepairState(0xc423808000, 0xc425660001, 0xc435022118, 0x8, 0xc4324b21f0, 0x3, 0x4, 0x0, 0x0, 0x0)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:528 +0x3a5
          github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendRestartMsg(0xc420140c00, 0x1b24da0, 0xc433ef1ea0)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3061 +0x1b61
          created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3003 +0xeba

          -------------------------------


          {code}

          logs:

          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.156.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.157.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.164.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.61.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.69.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.70.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.87.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.106.188.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.108.103.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.148.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.251.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.252.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.253.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.56.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.95.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.119.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.121.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.122.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.239.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.242.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.98.135.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.11.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.20.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.21.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.25.zip
          Build : 6.5.0-4471
          Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
          Scale : 3
          Iteration : 1st

          Step:
          {code:java}
          [2019-10-06T23:26:24-07:00, sequoiatools/eventing:6.5:cf2e4a] /eventing.py 172.23.98.135 8096 mad-hatter/bucket_op_n1ql.json Administrator password pause true
          warning using 'json' filter: unexpected end of JSON input []
          warning using 'json' filter: unexpected end of JSON input []
          {code}

          Pausing eventing handler started at 2019-10-06T23:26:24-07:00 and below logs taken at 2019-10-07 01:11:08,585.

          logs:

          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.156.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.157.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.164.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.5.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.61.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.67.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.69.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.70.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.87.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.104.88.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.106.100.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.106.188.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.108.103.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.148.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.251.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.252.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.253.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.96.95.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.119.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.121.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.122.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.239.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.97.242.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.98.135.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.99.11.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.99.21.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570434126/collectinfo-2019-10-07T074208-ns_1%40172.23.99.25.zip
          2019-10-07 01:11:08,585 - systestmon - INFO - ====== Log scan iteration number 39 complete. Sleeping for 3600 seconds ======
          girish.benakappa Girish Benakappa made changes -
          Link This issue Clones MB-36341 [ MB-36341 ]
          girish.benakappa Girish Benakappa made changes -
          Assignee Girish Benakappa [ girish.benakappa ] Jeelan Poola [ jeelan.poola ]
          jeelan.poola Jeelan Poola made changes -
          Assignee Jeelan Poola [ jeelan.poola ] Gautham Banasandra [ gautham.banasandra ]

          Girish Benakappa Which are the eventing nodes in the above list? Also, it would be great to create a supportal snapshot.

          jeelan.poola Jeelan Poola added a comment - Girish Benakappa Which are the eventing nodes in the above list? Also, it would be great to create a supportal snapshot.
          jeelan.poola Jeelan Poola added a comment - - edited

          Log file from node 172.23.98.135 is full of below logs

          2019-10-06T22:50:41.013-07:00 [Info] eventing-consumer [worker_bucket_op_n1ql_2:/tmp/127.0.0.1:8091_2_156534221.sock:7804] file : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/query/src/helper.cc line : 202 caller : GetErrorCodes : Returning empty valuefile : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/src/js_exception.cc line : 216 caller : CustomErrorCtor : Returning empty valuefile : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/src/js_exception.cc line : 182 caller : NewCustomError : Returning empty valueUnable to construct N1QLError : Unable to instantiate N1QLError
          2019-10-06T22:50:41.015-07:00 [Info] eventing-consumer [worker_bucket_op_n1ql_0:/tmp/127.0.0.1:8091_0_156534221.sock:7805] file : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/query/src/helper.cc line : 202 caller : GetErrorCodes : Returning empty valuefile : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/src/js_exception.cc line : 216 caller : CustomErrorCtor : Returning empty valuefile : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/src/js_exception.cc line : 182 caller : NewCustomError : Returning empty valueUnable to construct N1QLError : Unable to instantiate N1QLError
          2019-10-06T22:50:41.015-07:00 [Info] eventing-consumer [worker_bucket_op_n1ql_0:/tmp/127.0.0.1:8091_0_156534221.sock:7805] file : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/query/src/helper.cc line : 202 caller : GetErrorCodes : Returning empty valuefile : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/src/js_exception.cc line : 216 caller : CustomErrorCtor : Returning empty valuefile : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/src/js_exception.cc line : 182 caller : NewCustomError : Returning empty valueUnable to construct N1QLError : Unable to instantiate N1QLError
          

          Gautham Banasandra We need to fix above flood of logs. Else it will quickly overflow log file and we will lose ability to troubleshoot when this happens.

          The only instance of pause in log file began at

          2019-10-06T23:26:31.483-07:00 [Info] SuperSupervisor::SettingsChangeCallback [2] Function: bucket_op_n1ql current state: 2 requested status for deployment: true processing: false
          2019-10-06T23:26:31.483-07:00 [Info] SuperSupervisor::SettingsChangeCallback [2] Function: bucket_op_n1ql begin pausing process
          

          And completed at

          2019-10-06T23:29:32.551-07:00 [Info] Producer::Serve [bucket_op_n1ql:0] Closed stop chan and app log writer handle
          2019-10-06T23:29:32.551-07:00 [Info] SuperSupervisor::SettingsChangeCallback [2] Function: bucket_op_n1ql Cleaned up running Eventing.Producer instance
          2019-10-06T23:29:32.551-07:00 [Info] SuperSupervisor::SettingsChangeCallback [2] Function: bucket_op_n1ql pausing done
          

          Likely the logs have rolled over.

          jeelan.poola Jeelan Poola added a comment - - edited Log file from node 172.23.98.135 is full of below logs 2019-10-06T22:50:41.013-07:00 [Info] eventing-consumer [worker_bucket_op_n1ql_2:/tmp/127.0.0.1:8091_2_156534221.sock:7804] file : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/query/src/helper.cc line : 202 caller : GetErrorCodes : Returning empty valuefile : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/src/js_exception.cc line : 216 caller : CustomErrorCtor : Returning empty valuefile : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/src/js_exception.cc line : 182 caller : NewCustomError : Returning empty valueUnable to construct N1QLError : Unable to instantiate N1QLError 2019-10-06T22:50:41.015-07:00 [Info] eventing-consumer [worker_bucket_op_n1ql_0:/tmp/127.0.0.1:8091_0_156534221.sock:7805] file : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/query/src/helper.cc line : 202 caller : GetErrorCodes : Returning empty valuefile : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/src/js_exception.cc line : 216 caller : CustomErrorCtor : Returning empty valuefile : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/src/js_exception.cc line : 182 caller : NewCustomError : Returning empty valueUnable to construct N1QLError : Unable to instantiate N1QLError 2019-10-06T22:50:41.015-07:00 [Info] eventing-consumer [worker_bucket_op_n1ql_0:/tmp/127.0.0.1:8091_0_156534221.sock:7805] file : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/query/src/helper.cc line : 202 caller : GetErrorCodes : Returning empty valuefile : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/src/js_exception.cc line : 216 caller : CustomErrorCtor : Returning empty valuefile : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/libs/src/js_exception.cc line : 182 caller : NewCustomError : Returning empty valueUnable to construct N1QLError : Unable to instantiate N1QLError Gautham Banasandra We need to fix above flood of logs. Else it will quickly overflow log file and we will lose ability to troubleshoot when this happens. The only instance of pause in log file began at 2019-10-06T23:26:31.483-07:00 [Info] SuperSupervisor::SettingsChangeCallback [2] Function: bucket_op_n1ql current state: 2 requested status for deployment: true processing: false 2019-10-06T23:26:31.483-07:00 [Info] SuperSupervisor::SettingsChangeCallback [2] Function: bucket_op_n1ql begin pausing process And completed at 2019-10-06T23:29:32.551-07:00 [Info] Producer::Serve [bucket_op_n1ql:0] Closed stop chan and app log writer handle 2019-10-06T23:29:32.551-07:00 [Info] SuperSupervisor::SettingsChangeCallback [2] Function: bucket_op_n1ql Cleaned up running Eventing.Producer instance 2019-10-06T23:29:32.551-07:00 [Info] SuperSupervisor::SettingsChangeCallback [2] Function: bucket_op_n1ql pausing done Likely the logs have rolled over.
          vikas.chaudhary Vikas Chaudhary made changes -
          Priority Critical [ 2 ] Test Blocker [ 6 ]
          jeelan.poola Jeelan Poola added a comment -

          The symptoms and root cause here seem very similar to MB-36095 which is now fixed. The Toy with fix was tested against perf tests which reported the same issue in runs listed below successfully. Hence resolving it as duplicate. Please reopen if the issue is seen again.
          http://perf.jenkins.couchbase.com/job/themis-dev/898/console
          http://perf.jenkins.couchbase.com/job/themis-dev/899/console

          jeelan.poola Jeelan Poola added a comment - The symptoms and root cause here seem very similar to MB-36095 which is now fixed. The Toy with fix was tested against perf tests which reported the same issue in runs listed below successfully. Hence resolving it as duplicate. Please reopen if the issue is seen again. http://perf.jenkins.couchbase.com/job/themis-dev/898/console http://perf.jenkins.couchbase.com/job/themis-dev/899/console
          jeelan.poola Jeelan Poola made changes -
          Resolution Duplicate [ 3 ]
          Status Open [ 1 ] Resolved [ 5 ]

          Bulk closing all invalid, duplicate and won't fix bugs. Please feel free to reopen them

          raju Raju Suravarjjala added a comment - Bulk closing all invalid, duplicate and won't fix bugs. Please feel free to reopen them
          raju Raju Suravarjjala made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            Gautham.Banasandra Gautham Banasandra (Inactive)
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty