[MB-36341] [System Test] panic: runtime error: index out of range - Couchbase database

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 6.5.0
Fix Version/s: 6.5.0
Component/s: secondary-index
Labels:
- eagle-eye
- system-test
Environment:
component cluster

Triage:
Untriaged
Is this a Regression?:
Unknown

Description

Build : 6.5.0-4471
Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
Scale : 3
Iteration : 1st

Step:

[2019-10-04T12:26:40-07:00, sequoiatools/couchbase-cli:6.5:94d558] failover -c 172.23.108.103:8091 --server-failover 172.23.106.100:8091 -u Administrator -p password --force

[2019-10-04T12:27:26-07:00, sequoiatools/couchbase-cli:6.5:b16f29] rebalance -c 172.23.108.103:8091 -u Administrator -p password

warning using 'json' filter:  unexpected end of JSON input []

→

Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]

docker logs b16f29

docker start b16f29

*Unable to display progress bar on this os

JERROR: Rebalance failed. See logs for detailed reason. You can try again.

panic: (diag.log in 172.23.104.69 and 172.23.104.87)

2019-10-04T12:32:10.581-07:00, ns_log:0:info:message(ns_1@172.23.104.69) - Service 'indexer' exited with status 2. Restarting. Messages:

2019-10-04T12:32:09.130-07:00 [Info] default/default_claims/Backstore#17757113469883735399:0 Plasma: SMR reclaim pending is higher than expected: pending = 36 MB (expected = 10 MB), wCtxCnt = 30, changed reclaimList flush threshold from 12 to 13.

panic: runtime error: index out of range

goroutine 21707592 [running]:

panic(0xf8fe20, 0xc4200160d0)

        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1

github.com/couchbase/indexing/secondary/indexer.(*StreamState).updateRepairState(0xc42000fcc0, 0xc424430001, 0xc432425008, 0x8, 0xc4303a90f0, 0x3, 0x4, 0x0, 0x0, 0x0)

        goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:528 +0x3a5

github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendRestartMsg(0xc42001d280, 0x1b24da0, 0xc428205650)

        goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3061 +0x1b61

created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream

        goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3003 +0xeba

2019-10-04T12:32:10.766-07:00, ns_log:0:info:message(ns_1@172.23.104.87) - Service 'indexer' exited with status 2. Restarting. Messages:

2019-10-04T12:32:10.611-07:00 [Info] ServiceMgr::cleanupRebalanceRunning Cleanup

2019-10-04T12:32:10.611-07:00 [Info] ClustMgr:handleDelLocalValue Key RebalanceRunning

2019-10-04T12:32:10.612-07:00 [Info] Rebalancer::observeRebalance exiting err <nil>

panic: runtime error: index out of range

goroutine 34975387 [running]:

panic(0xf8fe20, 0xc4200160d0)

        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1

github.com/couchbase/indexing/secondary/indexer.(*StreamState).updateRepairState(0xc423808000, 0xc425660001, 0xc435022118, 0x8, 0xc4324b21f0, 0x3, 0x4, 0x0, 0x0, 0x0)

        goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:528 +0x3a5

github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendRestartMsg(0xc420140c00, 0x1b24da0, 0xc433ef1ea0)

        goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3061 +0x1b61

created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream

        goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3003 +0xeba

-------------------------------

logs:

Attachments

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

For Gerrit Dashboard: MB-36341
#	Subject	Branch	Project	Status	CR	V
116167,2	MB-36341 skip restartVb response for inactive stream	unstable	indexing	Status: MERGED	+2	+1
116168,3	MB-36341 use sessionId for restartVb	unstable	indexing	Status: MERGED	+2	+1

Activity

Ascending order - Click to sort in descending order

Girish Benakappa created issue - 04/Oct/19 3:39 PM

Girish Benakappa made changes - 04/Oct/19 3:39 PM

Field	Original Value	New Value
Link		This issue Clones ~~MB-36338~~ [ ~~MB-36338~~ ]

Girish Benakappa made changes - 04/Oct/19 3:39 PM

Component/s		secondary-index [ 11211 ]
Component/s	ns_server [ 10019 ]

Girish Benakappa made changes - 04/Oct/19 3:39 PM

Assignee

Ajit Yagaty [ ajit.yagaty ]

Jeelan Poola [ jeelan.poola ]

Girish Benakappa made changes - 04/Oct/19 3:43 PM

Description

Build : 6.5.0-4471
Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
Scale : 3
Iteration : 1st

Step:
{code:java}
[2019-10-04T09:44:15-07:00, sequoiatools/couchbase-cli:6.5:6a4039] server-add -c 172.23.108.103:8091 --server-add https://172.23.106.100 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
[2019-10-04T09:44:38-07:00, sequoiatools/couchbase-cli:6.5:b4a767] rebalance -c 172.23.108.103:8091 -u Administrator -p password
→

Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]

docker logs b4a767
docker start b4a767

*Unable to display progress bar on this os
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
[2019-10-04T09:55:07-07:00, sequoiatools/cmd:5a4414] 60
[2019-10-04T09:56:16-07:00, sequoiatools/cmd:348ec6] 300
{code}

Error: (diag.log in 172.23.108.103)

{code:java}
2019-10-04T09:54:52.724-07:00, compaction_daemon:0:warning:message(ns_1@172.23.97.119) - Compactor for view `mapreduce_view/default/_design/scale/replica` (pid [{type,
                                                                         view},
                                                                        {important,
                                                                         true},
                                                                        {name,
                                                                         <<"mapreduce_view/default/_design/scale/replica">>},
                                                                        {fa,
                                                                         {#Fun<compaction_daemon.23.62703019>,
                                                                          [<<"default">>,
                                                                           <<"_design/scale">>,
                                                                           mapreduce_view,
                                                                           replica,
                                                                           {config,
                                                                            {30,
                                                                             undefined},
                                                                            {30,
                                                                             undefined},
                                                                            undefined,
                                                                            false,
                                                                            false,
                                                                            {daemon_config,
                                                                             30,
                                                                             131072,
                                                                             20971520}},
                                                                           true,
                                                                           {[{type,
                                                                              bucket}]}]}}]) terminated unexpectedly: {updater_died,
                                                                                                                       {updater_error,
                                                                                                                        timeout}}
2019-10-04T09:54:52.930-07:00, ns_rebalancer:2:info:message(ns_1@172.23.108.103) - Bad replicators after rebalance:
Missing = [{'ns_1@172.23.108.103','ns_1@172.23.97.119',113},
           {'ns_1@172.23.108.103','ns_1@172.23.97.119',114},
           {'ns_1@172.23.108.103','ns_1@172.23.97.119',115},
           {'ns_1@172.23.108.103','ns_1@172.23.97.119',116},
           {'ns_1@172.23.108.103','ns_1@172.23.97.119',117},
           {'ns_1@172.23.108.103','ns_1@172.23.97.119',118},
           {'ns_1@172.23.108.103','ns_1@172.23.97.119',119},
           {'ns_1@172.23.108.103','ns_1@172.23.97.119',120},
           {'ns_1@172.23.108.103','ns_1@172.23.97.119',121}]
Extras = []
2019-10-04T09:54:52.933-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.108.103) - Rebalance exited with reason bad_replicas.
Rebalance Operation Id = 0b67a052f94895e5418ff35de8a74354
-------------------------------

per_node_processes('ns_1@172.23.108.103') =
     {<0.32074.207>,
      [{backtrace,
           [<<"Program counter: 0x00007f550eb437c8 (gen_server:loop/7 + 288)">>,
            <<"CP: 0x0000000000000000 (invalid)">>,<<"arity = 0">>,<<>>,
            <<"0x00007f538ccd6128 Return addr 0x00007f55553e6ff0 (proc_lib:init_p_do_apply/3 + 72)">>,
            <<"y(0) []">>,<<"y(1) infinity">>,
            <<"y(2) dcp_proxy">>,
            <<"(3) {state,#Port<0.236464>,{producer,\"replication:ns_1@172.23.106.100->ns_1@172.23.108.103:NEW_ORDER\",'ns_1@172.23.1">>,
            <<"y(4) <0.32074.207>">>,<<"y(5) <0.7577.208>">>,<<>>,
            <<"0x00007f538ccd6160 Return addr 0x0000000000942608 (<terminate process normally>)">>,
            <<"y(0) []">>,
            <<"y(1) Catch 0x00007f55553e7010 (proc_lib:init_p_do_apply/3 + 104)">>,
            <<>>]},
       {messages,[]},
       {dictionary,
           [{'$initial_call',{dcp_proxy,init,1}},
            {'$ancestors',
                ['dcp_replicator-NEW_ORDER-ns_1@172.23.106.100',
                 'dcp_sup-NEW_ORDER','single_bucket_kv_sup-NEW_ORDER',
                 ns_bucket_sup,ns_bucket_worker_sup,ns_server_sup,
                 ns_server_nodes_sup,<0.205.0>,ns_server_cluster_sup,root_sup,
                 <0.117.0>]}]},
       {registered_name,[]},
       {status,waiting},
       {initial_call,{proc_lib,init_p,5}},
       {error_handler,error_handler},
       {garbage_collection,
           [{max_heap_size,#{error_logger => true,kill => true,size => 0}},
            {min_bin_vheap_size,46422},
            {min_heap_size,233},
            {fullsweep_after,512},
            {minor_gcs,98}]},
       {garbage_collection_info,
           [{old_heap_block_size,6772},
            {heap_block_size,1598},
            {mbuf_size,0},
            {recent_size,24},
            {stack_size,10},
            {old_heap_size,2339},
            {heap_size,445},

{code}

logs:

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.156.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.164.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.61.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.87.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.56.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.95.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.242.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.99.25.zip

Girish Benakappa made changes - 04/Oct/19 3:43 PM

Link

This issue Clones ~~MB-36338~~ [ ~~MB-36338~~ ]

Deepkaran Salooja made changes - 04/Oct/19 3:47 PM

Assignee

Jeelan Poola [ jeelan.poola ]

Deepkaran Salooja [ deepkaran.salooja ]

Girish Benakappa made changes - 07/Oct/19 2:39 PM

Link

This issue is cloned by ~~MB-36356~~ [ ~~MB-36356~~ ]

Girish Benakappa made changes - 07/Oct/19 2:44 PM

Link

This issue is cloned by ~~MB-36356~~ [ ~~MB-36356~~ ]

Deepkaran Salooja made changes - 11/Oct/19 7:54 AM

Resolution		Fixed [ 1 ]
Status	Open [ 1 ]	Resolved [ 5 ]

Girish Benakappa made changes - 06/Nov/19 2:52 PM

Status

Resolved [ 5 ]

Closed [ 6 ]

People

Assignee:: Deepkaran Salooja

Reporter:: Girish Benakappa

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 04/Oct/19 3:39 PM

Updated:: 06/Nov/19 2:52 PM

Resolved:: 11/Oct/19 7:54 AM

Gerrit Reviews

There are no open Gerrit changes

Show There are 2 closed Gerrit changes

Hide There are 2 closed Gerrit changes

MB-36341 skip restartVb response for inactive stream: Gerrit Review:

MB-36341 use sessionId for restartVb: Gerrit Review:

Couchbase Server

[System Test] panic: runtime error: index out of range

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty