Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Multiple Errors and Failures

Description

CB version: Couchbase Server Community Edition 7.1.1 build 3175

We have tried below

  1. Tested with 3 node Cluster and then on a Single node cluster.

  2. We have increased the Node memory on single node
    2.1 Data: 14000 MiB
    2.2 Index: 6000 MiB
    2.3: Search: 6000 MiB

  3. Increased Disk IOPS from 300 to 3000

  4. We have open all ports on (in security group)

We have also tested this in Couchbase Server Enterprise Edition 7.1.1 build 3175 and everything works with no errors.

Components

Affects versions

Fix versions

Labels

Environment

Prod

Link to Log File, atop/blg, CBCollectInfo, Core dump

None

Release Notes Description

None

Activity

Show:

Sitaram Vemulapalli October 27, 2022 at 12:59 AM
Edited

From: xxxxxxxx
Date: Wednesday, October 26, 2022 at 9:46 AM

We have re-worked the 2 indexes which were mentioned in the initial email and the query seems to be working.

 

Did the same queries work before upgrade as well or are these new queries or indexes

>> Yes these queries worked before the upgrade and the indexes were the same.

Srinath Duvuru October 19, 2022 at 7:31 PM

The forum post says the "these errors have started popping in production after our migration to 7.1". We have not had changes to forestdb in a long time. Are the failing queries new queries or they used to work before and started to fail after migration?

 

Srinath Duvuru October 19, 2022 at 7:23 PM

From the e-mail

Hi all,
as requested, this is the index definition and we use it to fetch places nearby with some filters applied to it like the address, name, uid (the user id), public flag, cat( category of the place), etc.
 
CREATE INDEX idx_geo_location ON places (radians(location.lat), radians(location.`long`), location.lat, location.`long`, id, location, uid, bpid, brnchid, prntpid, address, name, `public`, openhs, closedhs, cat, del, moddt) WHERE tp = 'p' USING GSI;

Sitaram Vemulapalli October 19, 2022 at 1:06 PM

Please Add index definition and queries that use this index.

If the wider index is causing issue you should use the lean index (i.e reduce number of the index keys, by only have query predicate as part of the index) and let fetch the document(non-covered query). 

This is geo index. FTS has better option for these particular queries

 

Varun Velamuri October 11, 2022 at 1:03 AM

There are three panics in indexer logs in total.

Panic 1: Panic in FDB code - Probably due to corruption. CC: 

 

2022-10-02T10:04:04.988+00:00 [ERRO][FDB] doc_body checksum mismatch error in a database file '/couchbase-data/@2i/places_idx_geo_location_3744681898615285168_0.index/data.fdb.4' crc b6e15e1a != 467f364e (crc in doc) keylen 36361 metalen 0 bodylen 0 bodylen_ondisk 0 offset 99666582 fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x7f491f667345] runtime stack: runtime.throw({0x1110cae?, 0x0?})         /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/panic.go:992 +0x71 runtime.sigpanic()         /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/signal_unix.go:802 +0x3a9

 

Panic 2: Slice bounds out of range in scan_pipeline.go

 

panic: runtime error: slice bounds out of range [-24061:] [recovered]         panic: runtime error: slice bounds out of range [-24061:] goroutine 148469 [running]: github.com/couchbase/indexing/secondary/indexer.(*IndexScanSource).Routine.func1()         /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline.go:132 +0xf4 panic({0x1023160, 0xc009b02000})         /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/panic.go:838 +0x207 github.com/couchbase/indexing/secondary/indexer.projectKeys({0xc00e2e8000?, 0x10e70e9?, 0xb?}, {0x7f581fa12800, 0x83b, 0x83b}, {0xc00e2d5000, 0x0, 0x3800}, 0xc00e2c8000, ...)         /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline.go:762 +0x41d github.com/couchbase/indexing/secondary/indexer.(*IndexScanSource).Routine.func3({0x7f581fa12800?, 0x83b, 0x2c?})

Panic 3: Panic in ExplodeArray3

2022-10-02T10:20:58.576+00:00 [Fatal] ExplodeArray3:: recovered panic - runtime error: index out of range [0] with length 0         /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:312 +0x19 github.com/couchbase/indexing/secondary/collatejson.(*Codec).ExplodeArray3.func1()         /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/collatejson/array.go:118 +0xca panic({0x1023160, 0xc00c242558})         /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/panic.go:838 +0x207 github.com/couchbase/indexing/secondary/collatejson.(*Codec).code2n1ql(0xc00000ee10, {0x7fd26c16d7db, 0x1ab, 0x1ab}, {0xc00b7b9307, 0x0, 0x34f9}, 0x0)         /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/collatejson/collate.go:904 +0x6b7 github.com/couchbase/indexing/secondary/collatejson.(*Codec).code2n1ql(0xc00000ee10, {0x7fd26c16d7bd, 0x1c9, 0x1c9}, {0xc00b7b92ee, 0x0, 0x3512}, 0x0)         /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/collatejson/collate.go:943 +0x10d7 github.com/couchbase/indexing/secondary/collatejson.(*Codec).code2n1ql(0xc00000ee10, {0x7fd26c16d4ce, 0x4b8, 0x4b8}, {0xc00b7b9000, 0x0, 0x3800}, 0x0)

 

 

Won't Fix
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Is this a Regression?

Unknown

Triage

Untriaged

Operating System

Ubuntu 64-bit

Story Points

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created October 9, 2022 at 6:36 AM
Updated March 13, 2025 at 6:42 AM
Resolved March 13, 2025 at 6:42 AM
Instabug

Flag notifications