Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36897

Analytics node failing over when creating primary key indexes

    XMLWordPrintable

Details

    • Untriaged
    • Unknown
    • CX Sprint 177, CX Sprint 178, CX Sprint 179, CX Sprint 180, CX Sprint 183, CX Sprint 184, CX Sprint 185, CX Sprint 186, CX Sprint 187, CX Sprint 188, CX Sprint 189, CX Sprint 210, CX Sprint 211

    Attachments

      Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          michael.blow Michael Blow added a comment - - edited

          Hi Korrigan Clark, it seems reserving 15% of the quota for JVM native memory hasn't proven sufficient. Can you try running this same test reducing the analytics service quota on the nodes from what I'm assuming is 20GB to 18GB? Let's see if an additional 2GB of system memory provides enough headroom. (I'm downloading the logs but am on a slow connection at the moment)

          michael.blow Michael Blow added a comment - - edited Hi Korrigan Clark , it seems reserving 15% of the quota for JVM native memory hasn't proven sufficient. Can you try running this same test reducing the analytics service quota on the nodes from what I'm assuming is 20GB to 18GB? Let's see if an additional 2GB of system memory provides enough headroom. (I'm downloading the logs but am on a slow connection at the moment)

          this test is with 6io devices. Run with 18gb for analytics: http://perf.jenkins.couchbase.com/job/oceanus/2690/ 

          This also relates to this ticket: https://issues.couchbase.com/browse/MB-37519

          Will rerun with 12 iodevices and 18gb.

          korrigan.clark Korrigan Clark (Inactive) added a comment - this test is with 6io devices. Run with 18gb for analytics:  http://perf.jenkins.couchbase.com/job/oceanus/2690/   This also relates to this ticket:  https://issues.couchbase.com/browse/MB-37519 Will rerun with 12 iodevices and 18gb.
          korrigan.clark Korrigan Clark (Inactive) added a comment - with 12 iodevices and 18gb   http://perf.jenkins.couchbase.com/job/oceanus/2691/console https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-01-17T173850-ns_1%40172.23.96.205.zip https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-01-17T173850-ns_1%40172.23.96.5.zip https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-01-17T173850-ns_1%40172.23.96.57.zip https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-01-17T173850-ns_1%40172.23.96.7.zip https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-01-17T173850-ns_1%40172.23.96.8.zip https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-01-17T173850-ns_1%40172.23.96.9.zip

          Korrigan Clark, the issue causing the ingestion to fail has been fixed in build 7.0.0-2748. You can now run the volume tests with any number of iodevices and datasets. However, query 'TP02' needs to be rewritten similar to 'TP01' since the compiler currently does not generate an efficient plan for it, and the execution will take a very long time consuming resources and leading to a failure. It's a separate issue that we are aware of. Also, 'storageMaxActiveWritableDatasets' parameter is no longer supported, so set_max_active_writable_datasets should be removed from the test.

          ali.alsuliman Ali Alsuliman added a comment - Korrigan Clark , the issue causing the ingestion to fail has been fixed in build 7.0.0-2748. You can now run the volume tests with any number of iodevices and datasets. However, query 'TP02' needs to be rewritten similar to 'TP01' since the compiler currently does not generate an efficient plan for it, and the execution will take a very long time consuming resources and leading to a failure. It's a separate issue that we are aware of. Also, 'storageMaxActiveWritableDatasets' parameter is no longer supported, so set_max_active_writable_datasets should be removed from the test.

          I close this issue. We have a good run on build 7.0.0-4678.

          http://perf.jenkins.couchbase.com/job/oceanus-dev/113/ 

          bo-chun.wang Bo-Chun Wang added a comment - I close this issue. We have a good run on build 7.0.0-4678. http://perf.jenkins.couchbase.com/job/oceanus-dev/113/  

          People

            bo-chun.wang Bo-Chun Wang
            korrigan.clark Korrigan Clark (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty