Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36341

[System Test] panic: runtime error: index out of range

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Build : 6.5.0-4471
      Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Iteration : 1st

      Step:

      [2019-10-04T12:26:40-07:00, sequoiatools/couchbase-cli:6.5:94d558] failover -c 172.23.108.103:8091 --server-failover 172.23.106.100:8091 -u Administrator -p password --force
      [2019-10-04T12:27:26-07:00, sequoiatools/couchbase-cli:6.5:b16f29] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      warning using 'json' filter:  unexpected end of JSON input []
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
       
      docker logs b16f29
      docker start b16f29
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      

      panic: (diag.log in 172.23.104.69 and 172.23.104.87)

       
      2019-10-04T12:32:10.581-07:00, ns_log:0:info:message(ns_1@172.23.104.69) - Service 'indexer' exited with status 2. Restarting. Messages:
      2019-10-04T12:32:09.130-07:00 [Info] default/default_claims/Backstore#17757113469883735399:0 Plasma: SMR reclaim pending is higher than expected: pending = 36 MB (expected = 10 MB), wCtxCnt = 30, changed reclaimList flush threshold from 12 to 13.
      panic: runtime error: index out of range
       
      goroutine 21707592 [running]:
      panic(0xf8fe20, 0xc4200160d0)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1
      github.com/couchbase/indexing/secondary/indexer.(*StreamState).updateRepairState(0xc42000fcc0, 0xc424430001, 0xc432425008, 0x8, 0xc4303a90f0, 0x3, 0x4, 0x0, 0x0, 0x0)
              goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:528 +0x3a5
      github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendRestartMsg(0xc42001d280, 0x1b24da0, 0xc428205650)
              goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3061 +0x1b61
      created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream
              goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3003 +0xeba
       
      2019-10-04T12:32:10.766-07:00, ns_log:0:info:message(ns_1@172.23.104.87) - Service 'indexer' exited with status 2. Restarting. Messages:
      2019-10-04T12:32:10.611-07:00 [Info] ServiceMgr::cleanupRebalanceRunning Cleanup
      2019-10-04T12:32:10.611-07:00 [Info] ClustMgr:handleDelLocalValue Key RebalanceRunning
      2019-10-04T12:32:10.612-07:00 [Info] Rebalancer::observeRebalance exiting err <nil>
      panic: runtime error: index out of range
       
      goroutine 34975387 [running]:
      panic(0xf8fe20, 0xc4200160d0)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1
      github.com/couchbase/indexing/secondary/indexer.(*StreamState).updateRepairState(0xc423808000, 0xc425660001, 0xc435022118, 0x8, 0xc4324b21f0, 0x3, 0x4, 0x0, 0x0, 0x0)
              goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:528 +0x3a5
      github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendRestartMsg(0xc420140c00, 0x1b24da0, 0xc433ef1ea0)
              goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3061 +0x1b61
      created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream
              goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3003 +0xeba
       
      -------------------------------
       
      
      

      logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.156.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.164.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.61.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.87.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.56.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.95.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.20.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.25.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          girish.benakappa Girish Benakappa created issue -
          girish.benakappa Girish Benakappa made changes -
          Field Original Value New Value
          Link This issue Clones MB-36338 [ MB-36338 ]
          girish.benakappa Girish Benakappa made changes -
          Component/s secondary-index [ 11211 ]
          Component/s ns_server [ 10019 ]
          girish.benakappa Girish Benakappa made changes -
          Assignee Ajit Yagaty [ ajit.yagaty ] Jeelan Poola [ jeelan.poola ]
          girish.benakappa Girish Benakappa made changes -
          Description Build : 6.5.0-4471
          Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
          Scale : 3
          Iteration : 1st

          Step:
          {code:java}
          [2019-10-04T09:44:15-07:00, sequoiatools/couchbase-cli:6.5:6a4039] server-add -c 172.23.108.103:8091 --server-add https://172.23.106.100 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
          [2019-10-04T09:44:38-07:00, sequoiatools/couchbase-cli:6.5:b4a767] rebalance -c 172.23.108.103:8091 -u Administrator -p password


          Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]

          docker logs b4a767
          docker start b4a767

          *Unable to display progress bar on this os
          JERROR: Rebalance failed. See logs for detailed reason. You can try again.
          [2019-10-04T09:55:07-07:00, sequoiatools/cmd:5a4414] 60
          [2019-10-04T09:56:16-07:00, sequoiatools/cmd:348ec6] 300
          {code}


          Error: (diag.log in 172.23.108.103)

          {code:java}
          2019-10-04T09:54:52.724-07:00, compaction_daemon:0:warning:message(ns_1@172.23.97.119) - Compactor for view `mapreduce_view/default/_design/scale/replica` (pid [{type,
                                                                                   view},
                                                                                  {important,
                                                                                   true},
                                                                                  {name,
                                                                                   <<"mapreduce_view/default/_design/scale/replica">>},
                                                                                  {fa,
                                                                                   {#Fun<compaction_daemon.23.62703019>,
                                                                                    [<<"default">>,
                                                                                     <<"_design/scale">>,
                                                                                     mapreduce_view,
                                                                                     replica,
                                                                                     {config,
                                                                                      {30,
                                                                                       undefined},
                                                                                      {30,
                                                                                       undefined},
                                                                                      undefined,
                                                                                      false,
                                                                                      false,
                                                                                      {daemon_config,
                                                                                       30,
                                                                                       131072,
                                                                                       20971520}},
                                                                                     true,
                                                                                     {[{type,
                                                                                        bucket}]}]}}]) terminated unexpectedly: {updater_died,
                                                                                                                                 {updater_error,
                                                                                                                                  timeout}}
          2019-10-04T09:54:52.930-07:00, ns_rebalancer:2:info:message(ns_1@172.23.108.103) - Bad replicators after rebalance:
          Missing = [{'ns_1@172.23.108.103','ns_1@172.23.97.119',113},
                     {'ns_1@172.23.108.103','ns_1@172.23.97.119',114},
                     {'ns_1@172.23.108.103','ns_1@172.23.97.119',115},
                     {'ns_1@172.23.108.103','ns_1@172.23.97.119',116},
                     {'ns_1@172.23.108.103','ns_1@172.23.97.119',117},
                     {'ns_1@172.23.108.103','ns_1@172.23.97.119',118},
                     {'ns_1@172.23.108.103','ns_1@172.23.97.119',119},
                     {'ns_1@172.23.108.103','ns_1@172.23.97.119',120},
                     {'ns_1@172.23.108.103','ns_1@172.23.97.119',121}]
          Extras = []
          2019-10-04T09:54:52.933-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.108.103) - Rebalance exited with reason bad_replicas.
          Rebalance Operation Id = 0b67a052f94895e5418ff35de8a74354
          -------------------------------


          per_node_processes('ns_1@172.23.108.103') =
               {<0.32074.207>,
                [{backtrace,
                     [<<"Program counter: 0x00007f550eb437c8 (gen_server:loop/7 + 288)">>,
                      <<"CP: 0x0000000000000000 (invalid)">>,<<"arity = 0">>,<<>>,
                      <<"0x00007f538ccd6128 Return addr 0x00007f55553e6ff0 (proc_lib:init_p_do_apply/3 + 72)">>,
                      <<"y(0) []">>,<<"y(1) infinity">>,
                      <<"y(2) dcp_proxy">>,
                      <<"(3) {state,#Port<0.236464>,{producer,\"replication:ns_1@172.23.106.100->ns_1@172.23.108.103:NEW_ORDER\",'ns_1@172.23.1">>,
                      <<"y(4) <0.32074.207>">>,<<"y(5) <0.7577.208>">>,<<>>,
                      <<"0x00007f538ccd6160 Return addr 0x0000000000942608 (<terminate process normally>)">>,
                      <<"y(0) []">>,
                      <<"y(1) Catch 0x00007f55553e7010 (proc_lib:init_p_do_apply/3 + 104)">>,
                      <<>>]},
                 {messages,[]},
                 {dictionary,
                     [{'$initial_call',{dcp_proxy,init,1}},
                      {'$ancestors',
                          ['dcp_replicator-NEW_ORDER-ns_1@172.23.106.100',
                           'dcp_sup-NEW_ORDER','single_bucket_kv_sup-NEW_ORDER',
                           ns_bucket_sup,ns_bucket_worker_sup,ns_server_sup,
                           ns_server_nodes_sup,<0.205.0>,ns_server_cluster_sup,root_sup,
                           <0.117.0>]}]},
                 {registered_name,[]},
                 {status,waiting},
                 {initial_call,{proc_lib,init_p,5}},
                 {error_handler,error_handler},
                 {garbage_collection,
                     [{max_heap_size,#{error_logger => true,kill => true,size => 0}},
                      {min_bin_vheap_size,46422},
                      {min_heap_size,233},
                      {fullsweep_after,512},
                      {minor_gcs,98}]},
                 {garbage_collection_info,
                     [{old_heap_block_size,6772},
                      {heap_block_size,1598},
                      {mbuf_size,0},
                      {recent_size,24},
                      {stack_size,10},
                      {old_heap_size,2339},
                      {heap_size,445},

          {code}

          logs:

          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.156.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.157.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.164.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.61.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.67.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.69.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.70.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.87.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.106.100.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.106.188.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.108.103.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.148.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.251.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.252.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.253.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.56.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.95.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.119.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.121.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.122.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.239.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.242.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.98.135.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.99.20.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.99.21.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.99.25.zip
          Build : 6.5.0-4471
          Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
          Scale : 3
          Iteration : 1st

          Step:
          {code:java}
          [2019-10-04T12:26:40-07:00, sequoiatools/couchbase-cli:6.5:94d558] failover -c 172.23.108.103:8091 --server-failover 172.23.106.100:8091 -u Administrator -p password --force
          [2019-10-04T12:27:26-07:00, sequoiatools/couchbase-cli:6.5:b16f29] rebalance -c 172.23.108.103:8091 -u Administrator -p password
          warning using 'json' filter: unexpected end of JSON input []


          Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]

          docker logs b16f29
          docker start b16f29

          *Unable to display progress bar on this os
          JERROR: Rebalance failed. See logs for detailed reason. You can try again.
          {code}


          panic: (diag.log in 172.23.104.69 and 172.23.104.87)
          {code:java}

          2019-10-04T12:32:10.581-07:00, ns_log:0:info:message(ns_1@172.23.104.69) - Service 'indexer' exited with status 2. Restarting. Messages:
          2019-10-04T12:32:09.130-07:00 [Info] default/default_claims/Backstore#17757113469883735399:0 Plasma: SMR reclaim pending is higher than expected: pending = 36 MB (expected = 10 MB), wCtxCnt = 30, changed reclaimList flush threshold from 12 to 13.
          panic: runtime error: index out of range

          goroutine 21707592 [running]:
          panic(0xf8fe20, 0xc4200160d0)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1
          github.com/couchbase/indexing/secondary/indexer.(*StreamState).updateRepairState(0xc42000fcc0, 0xc424430001, 0xc432425008, 0x8, 0xc4303a90f0, 0x3, 0x4, 0x0, 0x0, 0x0)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:528 +0x3a5
          github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendRestartMsg(0xc42001d280, 0x1b24da0, 0xc428205650)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3061 +0x1b61
          created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3003 +0xeba

          2019-10-04T12:32:10.766-07:00, ns_log:0:info:message(ns_1@172.23.104.87) - Service 'indexer' exited with status 2. Restarting. Messages:
          2019-10-04T12:32:10.611-07:00 [Info] ServiceMgr::cleanupRebalanceRunning Cleanup
          2019-10-04T12:32:10.611-07:00 [Info] ClustMgr:handleDelLocalValue Key RebalanceRunning
          2019-10-04T12:32:10.612-07:00 [Info] Rebalancer::observeRebalance exiting err <nil>
          panic: runtime error: index out of range

          goroutine 34975387 [running]:
          panic(0xf8fe20, 0xc4200160d0)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1
          github.com/couchbase/indexing/secondary/indexer.(*StreamState).updateRepairState(0xc423808000, 0xc425660001, 0xc435022118, 0x8, 0xc4324b21f0, 0x3, 0x4, 0x0, 0x0, 0x0)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:528 +0x3a5
          github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendRestartMsg(0xc420140c00, 0x1b24da0, 0xc433ef1ea0)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3061 +0x1b61
          created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3003 +0xeba

          -------------------------------


          {code}

          logs:

          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.156.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.157.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.164.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.61.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.69.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.70.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.104.87.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.106.188.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.108.103.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.148.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.251.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.252.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.253.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.56.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.96.95.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.119.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.121.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.122.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.239.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.97.242.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.98.135.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.11.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.20.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.21.zip
          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570217693/collectinfo-2019-10-04T193455-ns_1%40172.23.99.25.zip
          girish.benakappa Girish Benakappa made changes -
          Link This issue Clones MB-36338 [ MB-36338 ]
          deepkaran.salooja Deepkaran Salooja made changes -
          Assignee Jeelan Poola [ jeelan.poola ] Deepkaran Salooja [ deepkaran.salooja ]
          girish.benakappa Girish Benakappa made changes -
          Link This issue is cloned by MB-36356 [ MB-36356 ]
          girish.benakappa Girish Benakappa made changes -
          Link This issue is cloned by MB-36356 [ MB-36356 ]

          Bucket got removed from the stream state

           
          2019-10-04T12:32:06.671-07:00 [Info] Timekeeper::handleRemoveBucketFromStream
                  Message: MsgStreamUpdate
                  Type: REMOVE_BUCKET_FROM_STREAM
                  Stream: MAINT_STREAM
                  Bucket: DISTRICT
                  BuildTS: []
                  IndexList: []
                  Async: false
                  SessionId: 1
                  RestartTs: <nil>
          2019-10-04T12:32:06.671-07:00 [Info] Timekeeper::stopTimer MAINT_STREAM DISTRICT
          2019-10-04T12:32:06.671-07:00 [Info] StreamState::cleanupBucketFromStream Bucket DISTRICT Deleted from Stream MAINT_STREAM 
          
          

          But the repair is still outstanding

           
          panic: runtime error: index out of range
           
          goroutine 21707592 [running]:
          panic(0xf8fe20, 0xc4200160d0)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1
          github.com/couchbase/indexing/secondary/indexer.(*StreamState).updateRepairState(0xc42000fcc0, 0xc424430001, 0xc432425008, 0x8, 0xc4303a90f0, 0x3, 0x4, 0x0, 0x0, 0x0)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:528 +0x3a5
          github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendRestartMsg(0xc42001d280, 0x1b24da0, 0xc428205650)
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3061 +0x1b61
          created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream
                  goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3003 +0xeba
          
          

          deepkaran.salooja Deepkaran Salooja added a comment - Bucket got removed from the stream state   2019-10-04T12:32:06.671-07:00 [Info] Timekeeper::handleRemoveBucketFromStream Message: MsgStreamUpdate Type: REMOVE_BUCKET_FROM_STREAM Stream: MAINT_STREAM Bucket: DISTRICT BuildTS: [] IndexList: [] Async: false SessionId: 1 RestartTs: <nil> 2019-10-04T12:32:06.671-07:00 [Info] Timekeeper::stopTimer MAINT_STREAM DISTRICT 2019-10-04T12:32:06.671-07:00 [Info] StreamState::cleanupBucketFromStream Bucket DISTRICT Deleted from Stream MAINT_STREAM But the repair is still outstanding   panic: runtime error: index out of range   goroutine 21707592 [running]: panic(0xf8fe20, 0xc4200160d0) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1 github.com/couchbase/indexing/secondary/indexer.(*StreamState).updateRepairState(0xc42000fcc0, 0xc424430001, 0xc432425008, 0x8, 0xc4303a90f0, 0x3, 0x4, 0x0, 0x0, 0x0) goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:528 +0x3a5 github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendRestartMsg(0xc42001d280, 0x1b24da0, 0xc428205650) goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3061 +0x1b61 created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3003 +0xeba
          deepkaran.salooja Deepkaran Salooja made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]

          Build couchbase-server-6.5.0-4565 contains indexing commit f63d364 with commit message:
          MB-36341 use sessionId for restartVb

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4565 contains indexing commit f63d364 with commit message: MB-36341 use sessionId for restartVb

          Build couchbase-server-6.5.0-4565 contains indexing commit 18bb20a with commit message:
          MB-36341 skip restartVb response for inactive stream

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4565 contains indexing commit 18bb20a with commit message: MB-36341 skip restartVb response for inactive stream

          Build couchbase-server-7.0.0-1010 contains indexing commit f63d364 with commit message:
          MB-36341 use sessionId for restartVb

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1010 contains indexing commit f63d364 with commit message: MB-36341 use sessionId for restartVb

          Build couchbase-server-7.0.0-1010 contains indexing commit 18bb20a with commit message:
          MB-36341 skip restartVb response for inactive stream

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1010 contains indexing commit 18bb20a with commit message: MB-36341 skip restartVb response for inactive stream

          Did not find this issue in gsi and longevity system test with 6.5.0-4676, hence closing the issue.

          girish.benakappa Girish Benakappa added a comment - Did not find this issue in gsi and longevity system test with 6.5.0-4676, hence closing the issue.
          girish.benakappa Girish Benakappa made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            deepkaran.salooja Deepkaran Salooja
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty