Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54200

Plasma crash is observed in Elixir cluster. Service 'index' exited with status 2. Restarting. Messages: github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insertSecArrayIndex

    XMLWordPrintable

Details

    Description

      It is kind of tough to come up with the exact steps to reproduce it as this is a shared environment. All i have is the logs to debug what's going on.

      Loading sample bucket 5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5 failed. Samples loader exited with status 1.
      		 
      Loader's output was:
      		 
       
      		 
      2022-10-19T16:49:06.887+00:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools'
      		 
      2022-10-19T16:49:06.888+00:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools'
      		 
      2022-10-19T16:49:06.888+00:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default'
      		 
      2022-10-19T16:49:06.895+00:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default'
      		 
      2022-10-19T16:49:06.895+00:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets'
      		 
      2022-10-19T16:49:06.904+00:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets'
      		 
      2022-10-19T16:49:06.904+00:00 (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"7e7489f28b7eac90a26238d9a16b70a3","developer_preview":false,"version":{"min_version":"7.5.0","is_mixed_cluster":false},"max_vbuckets":64,"uniform_vbuckets":true}
      		 
      2022-10-19T16:49:06.975+00:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:06.981+00:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:06.981+00:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5/scopes'
      		 
      2022-10-19T16:49:06.981+00:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5/scopes'
      		 
      2022-10-19T16:49:06.981+00:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5/scopes/@ensureManifest/1'
      		 
      2022-10-19T16:49:06.984+00:00 (REST) (Attempt 1) (POST) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5/scopes/@ensureManifest/1'
      		 
      2022-10-19T16:49:06.984+00:00 (Plan) (Bucket) Creating remote bucket '5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5' if it does not exist
      		 
      2022-10-19T16:49:06.984+00:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:06.988+00:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:06.988+00:00 (Plan) (Bucket) Successfully ensured bucket '5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5' exists | {"number":1,"duration":"4.227541ms"}
      		 
      2022-10-19T16:49:06.988+00:00 (Plan) (Bucket) Automatically creating/editing scopes/collections for bucket '5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:06.988+00:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:06.992+00:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:06.992+00:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5/scopes'
      		 
      2022-10-19T16:49:06.993+00:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5/scopes'
      		 
      2022-10-19T16:49:06.993+00:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5/scopes/@ensureManifest/1'
      		 
      2022-10-19T16:49:06.995+00:00 (REST) (Attempt 1) (POST) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5/scopes/@ensureManifest/1'
      		 
      2022-10-19T16:49:06.995+00:00 (REST) (Attempt 1) (PUT) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5/scopes?validOnUid=1'
      		 
      2022-10-19T16:49:07.036+00:00 (REST) (Attempt 1) (PUT) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5/scopes?validOnUid=1'
      		 
      2022-10-19T16:49:07.036+00:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5/scopes/@ensureManifest/2'
      		 
      2022-10-19T16:49:07.076+00:00 (REST) (Attempt 1) (POST) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5/scopes/@ensureManifest/2'
      		 
      2022-10-19T16:49:07.076+00:00 (Plan) (Bucket) Successfully created/edited scopes/collections for bucket '5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5' | {"number":2,"duration":"87.570323ms"}
      		 
      2022-10-19T16:49:07.076+00:00 (Plan) (Bucket) Transferring bucket configuration for bucket '5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:07.076+00:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:07.080+00:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:07.080+00:00 Editing bucket to conform with settings provided
      		 
      2022-10-19T16:49:07.080+00:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:07.110+00:00 (REST) (Attempt 1) (POST) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:07.110+00:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:07.115+00:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://c3hqwfbcg9rsw32z.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:8091/pools/default/buckets/5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:07.115+00:00 (Plan) (Bucket) Successfully transferred bucket configuration for bucket '5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5' | {"number":3,"duration":"38.852482ms"}
      		 
      2022-10-19T16:49:07.115+00:00 (Plan) (Index) Transferring views definitions for bucket '5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:07.192+00:00 (Plan) (Index) Successfully transferred views definitions for bucket '5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5' | {"number":4,"duration":"77.372823ms"}
      		 
      2022-10-19T16:49:07.192+00:00 (Plan) (Query) Executing queries for bucket '5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5'
      		 
      2022-10-19T16:49:07.277+00:00 (Couchbase) Creating Query connection to cluster 'couchbase://fznz4zdh-rwwpyd.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:11210,ibzhmguucyxzgipp.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:11210,x6gipfeqo3unwmi.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:11210'
      		 
      2022-10-19T16:49:07.342+00:00 (Couchbase) Successfully created Query connection to cluster 'couchbase://fznz4zdh-rwwpyd.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:11210,ibzhmguucyxzgipp.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:11210,x6gipfeqo3unwmi.3eja7xznih9um94b.sandbox.nonprod-project-avengers.com:11210'
      		 
      2022-10-19T16:49:13.104+00:00 JSON import failed: failed to execute cluster operations: failed to execute bucket operation for bucket '5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5': failed to execute N1QL queries for bucket '5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5': failed to execute queries against sink bucket: failed to execute query: internal server failure | {"statement":"BUILD INDEX ON `5a89c9b7-02f0-4267-9a78-a6e2d08e8005-7l9hm5`.`inventory`.`route`((SELECT RAW name FROM system:indexes WHERE keyspace_id = 'route' AND state = 'deferred'))","client_context_id":"ef18652e-4d60-4f23-8cd3-eee37a7780e9","errors":[{"code":5000,"message":"BuildIndexes - cause: Build index fails. Some index will be retried building in the background. For more details, please check index status.\n"}],"http_status_code":500}
      		 
      JSON import failed: internal server failure

      followed by a crash:

      Service 'index' exited with status 2. Restarting. Messages:
      		 
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insertSecArrayIndex(0x401cbfd600, {0x4010d51c20, 0x9a, 0xa0}, {0x4013fd1170, 0xb, 0x10}, 0x7, 0x0, 0x4014136980)
      		 
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:1283 +0x1934
      		 
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insert(0x401cbfd600, {0x4010d51c20?, 0x4014332678?, 0x0?}, {0x4013fd1170?, 0x3f00400113d601?, 0x5?}, 0x8?, 0x57?, 0x3b?)
      		 
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:880 +0x70
      		 
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker(0x401cbfd600, 0x7)
      		 
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:755 +0x2f4
      		 
      created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initWriters
      		 
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:3450 +0x368

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-54204 Service 'index' exited with status 2. couchbase/indexing/secondary/indexer.(*indexer).handleStorageWarmupDone while deleting a bucket during index building.

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty