Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
Build: 7.0.0-4291
Panic found in : test_recovery_disk_snapshot with OSO enabled
Test: ./testrunner -i /tmp/testexec.14321.ini -p reset_services=True -t gsi.recovery_gsi.CollectionsSecondaryIndexingRecoveryTests.test_recovery_disk_snapshot,default_bucket=false,defer_build=False,java_sdk_client=True,nodes_init=3,services_init=kv-kv:n1ql-index,all_collections=True,num_items_in_collection=10000,percent_update=30,percent_delete=10,reset_services=True,enable_oso=True
- setup cluster with 3 nodes: kv, kv+n1ql, index
- create 2 scopes and 2 collections
- set enableOSO to True
- create index on each of the collection
- set indexer.settings.persisted_snapshot.moi.interval to 3000
- Load 10000 docs on each collection
- wait for mutation processing
- set {"indexer.settings.persisted_snapshot.moi.interval": 1200000}
- load another 10000 docs on each collection
- wait for mutation processing
- kill indexer
- Validate if indexes are recovering from snapshot.
It seems panic found in the log while creating index on index node:
[2021-01-25 11:33:16,348] - [tuq_helper:287] INFO - RUN QUERY CREATE INDEX idx_test_scope_2_test_collection_2job_title0 ON default:test_bucket.test_scope_2.test_collection_2(job_title) WHERE job_title IS NOT NULL USING GSI WITH {'defer_build': False}
|
[2021-01-25 11:33:16,357] - [rest_client:3872] INFO - query params : statement=CREATE+INDEX+idx_test_scope_2_test_collection_2job_title0+ON+default%3Atest_bucket.test_scope_2.test_collection_2%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D
|
[2021-01-25 11:33:48,006] - [tuq_helper:310] INFO - TOTAL ELAPSED TIME: 31.641772102s
|
Panic found in multiple places, first instance is below:
2021-01-25T11:33:16.553-08:00 [Info] TK StreamBegin MAINT_STREAM test_bucket 0 32563383099164 3 1
|
2021-01-25T11:33:16.553-08:00 [Info] TK SystemEvent MAINT_STREAM test_bucket 0 32563383099164 4 1 12 4
|
2021-01-25T11:33:16.553-08:00 [Info] TK SystemEvent MAINT_STREAM test_bucket 0 32563383099164 5 1 9 5
|
2021-01-25T11:33:16.553-08:00 [Info] TK SystemEvent MAINT_STREAM test_bucket 0 32563383099164 6 1 9 6
|
panic: runtime error: invalid memory address or nil pointer dereference
|
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb5 pc=0xf2ea1f]
|
|
|
goroutine 29 [running]:
|
github.com/couchbase/indexing/secondary/common.(*TsVbuuid).HasOpenOSOSnap(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/timestamp.go:301
|
github.com/couchbase/indexing/secondary/indexer.(*timekeeper).checkInitStreamReadyToMerge(0xc000186580, 0xc005570003, 0xc005ed3e00, 0x2a, 0x0, 0xc0049b8810)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:2331 +0x14f
|
github.com/couchbase/indexing/secondary/indexer.(*timekeeper).checkPendingStreamMerge.func1(0xc005ed0003, 0xc005ed3e00, 0x2a)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:4074 +0x14e
|
github.com/couchbase/indexing/secondary/indexer.(*timekeeper).checkPendingStreamMerge(0xc000186580, 0xc004df0001, 0xc005ed2f00, 0xb)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:4086 +0x23b
|
github.com/couchbase/indexing/secondary/indexer.(*timekeeper).handleFlushDoneMaintStream(0xc000186580, 0x145f560, 0xc0049b87b0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:758 +0x214
|
github.com/couchbase/indexing/secondary/indexer.(*timekeeper).handleFlushDone(0xc000186580, 0x145f560, 0xc0049b87b0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:663 +0x434
|
github.com/couchbase/indexing/secondary/indexer.(*timekeeper).handleSupervisorCommands(0xc000186580, 0x145f560, 0xc0049b87b0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:191 +0x2e4
|
github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0xc000186580)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:128 +0x56
|
created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:101 +0x12b
|
2021-01-25T11:33:16.725-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=d82ac5205f89a9971567cef8a6843d86 -ipv6=false -isEnterprise=true]
|
2021-01-25T11:33:16.734-08:00 [Info] Indexer::NewIndexer Status Warmup
|
2021-01-25T11:33:16.736-08:00 [Info] Setting buffer block size to 16384 bytes
|
|
Logs attached: 172.23.100.21 is the index node.