Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46464

Indexer runtime: out of memory

    XMLWordPrintable

Details

    Description

      Build: 7.0.0 build 5219

      Steps:

      • Multi-node cluster with KV, N1ql+index, FTS
      • 3 Couchbase buckets (Each bucket with 3 scopes and 2 collections under each scope. Total 18 collections on 3 buckets)
      • Creating 50 GSI on each available collection with gsi_index_replica=1

      Observation:

      1. On bucket "6RQl-30-113000" 150 total GSI indexes are created
      2. On bucket  "Eu0JTbzu37-30-110000" scope "bH0NXefmzmkbkYBCuf-30-112000" all 100 GSIs are created
      3. While trying to create indexes on "Eu0JTbzu37-30-110000" scope "default", indexer crashed with exit status 2
      4. Service 'indexer' exited with status 2. Restarting. Messages:
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x8f
        github.com/couchbase/indexing/secondary/indexer.(*kvSender).openMutationStream(0xc0041a6cf0, 0xc0022b0003, 0xc03d3d5da0, 0x14, 0x210dfd0, 0x1, 0xc06b457400, 0x1, 0x1, 0x0, ...)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:396 +0x8a4
        created by github.com/couchbase/indexing/secondary/indexer.(*kvSender).handleOpenStream
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:171 +0x267
         
        goroutine 618624 [chan receive]:
        github.com/couchbase/plasma.(*deallocCtx).smrWorker(0xc060f43d40)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/smr.go:323 +0xea
        created by github.com/couchbase/plasma.(*deallocCtx).startSMRWorker
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/smr.go:315 +0xc5

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          jliang John Liang added a comment - - edited

          Ashwin Govindarajulu Can you provide the following info?
          1) the number of indexes this test has for Mad-hatter
          2) the number of collection indexes as well as the number of non-collection index (on default scope/collection) in CC
          3) do you use the same cluster size + memory quota + cpu for mad-hatter and CC test?

          jliang John Liang added a comment - - edited Ashwin Govindarajulu Can you provide the following info? 1) the number of indexes this test has for Mad-hatter 2) the number of collection indexes as well as the number of non-collection index (on default scope/collection) in CC 3) do you use the same cluster size + memory quota + cpu for mad-hatter and CC test?
          ashwin.govindarajulu Ashwin Govindarajulu added a comment - - edited

          1) the number of indexes this test has for Mad-hatter --> 200 active GSI
          2) the number of collection indexes as well as the number of non-collection index (on default scope/collection) in CC --> In the above case, I am trying to create 600 GSIs in total. The crash is seen somewhere after creating 300-350 Indexes
          3) do you use the same cluster size + memory quota + cpu for mad-hatter and CC test? --> This test was originally run on 16G RAM boxes in MH testing. But the current run where we are hitting this issue was done on 4G boxes.

          Update:

          Ran the same scenario on 16G boxes again with 4G indexer memory quota. Able to create all 300 required indexes without issue.

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - - edited 1) the number of indexes this test has for Mad-hatter --> 200 active GSI 2) the number of collection indexes as well as the number of non-collection index (on default scope/collection) in CC --> In the above case, I am trying to create 600 GSIs in total. The crash is seen somewhere after creating 300-350 Indexes 3) do you use the same cluster size + memory quota + cpu for mad-hatter and CC test? --> This test was originally run on 16G RAM boxes in MH testing. But the current run where we are hitting this issue was done on 4G boxes. Update : Ran the same scenario on 16G boxes again with 4G indexer memory quota. Able to create all 300 required indexes without issue.
          jliang John Liang added a comment -

          >>This test was originally run on 16G RAM boxes in MH testing. But the current run where we are hitting this issue was done on 4G boxes.
          Ashwin Govindarajulu It is not just the just size of the box that matters. What is the indexer memory quota that you use to run on MH for 200 GSI indexes?

          >> Ran the same scenario on 16G boxes again with 4G indexer memory quota. Able to create all 300 required indexes without issue.
          Ashwin Govindarajulu For 16G boxes what is the indexer memory quota that you use?

          jliang John Liang added a comment - >>This test was originally run on 16G RAM boxes in MH testing. But the current run where we are hitting this issue was done on 4G boxes. Ashwin Govindarajulu It is not just the just size of the box that matters. What is the indexer memory quota that you use to run on MH for 200 GSI indexes? >> Ran the same scenario on 16G boxes again with 4G indexer memory quota. Able to create all 300 required indexes without issue. Ashwin Govindarajulu For 16G boxes what is the indexer memory quota that you use?
          ashwin.govindarajulu Ashwin Govindarajulu added a comment - - edited

          What is the indexer memory quota that you use to run on MH for 200 GSI indexes? 

          I think I had set 2G indexer quota for Mad-hatter testing. (I am not fully sure on this since the cbcollect logs got deleted from the aws bucket)

          For 16G boxes what is the indexer memory quota that you use?

          For this I have set 4G of Index memory quota.

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - - edited What is the indexer memory quota that you use to run on MH for 200 GSI indexes?  I think I had set 2G indexer quota for Mad-hatter testing. (I am not fully sure on this since the cbcollect logs got deleted from the aws bucket) For 16G boxes what is the indexer memory quota that you use? For this I have set 4G of Index memory quota.
          jliang John Liang added a comment -

          Ashwin Govindarajulu Based on the comment from Deep, the test will require at least 4G of RAM just for the couchbase processes, not to mention RAM used by OS. So it wouldn't work with a 4G box. Since you also mention that you use a 16G box in MH and you no longer have the issue after switching to 16G box in CC, I'll close this issue.

          In CC, there is more overhead per non-collection index. In general, customers would have more memory with hundreds of indexes. I'll file a separate improvement for this.

          jliang John Liang added a comment - Ashwin Govindarajulu Based on the comment from Deep, the test will require at least 4G of RAM just for the couchbase processes, not to mention RAM used by OS. So it wouldn't work with a 4G box. Since you also mention that you use a 16G box in MH and you no longer have the issue after switching to 16G box in CC, I'll close this issue. In CC, there is more overhead per non-collection index. In general, customers would have more memory with hundreds of indexes. I'll file a separate improvement for this.

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty