Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29028

Query REST API retries cause ns_server memory to dramatically spike (was: Service 'ns_server' exited with status 137 observed during centos longevity test)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.5.0
    • 5.5.0
    • query
    • Centos cluster

    Description

      Build : 5.5.0-2361
      We observe Internal server error , after we failover 2 node in centos longevity test. 

      step

      [2018-03-31T09:56:05-07:00, sequoiatools/couchbase-cli:9d514c] server-add -c 172.23.108.103:8091 --server-add 172.23.108.104:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      		 
      [2018-03-31T09:58:43-07:00, sequoiatools/couchbase-cli:a2eaa1] failover -c 172.23.108.103:8091 --server-failover 172.23.96.56:8091 -u Administrator -p password
      		 
      [2018-03-31T10:19:43-07:00, sequoiatools/couchbase-cli:207967] failover -c 172.23.108.103:8091 --server-failover 172.23.97.238:8091 -u Administrator -p password --force
      		 
      [2018-03-31T10:20:00-07:00, sequoiatools/couchbase-cli:9c1a88] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      		 
      →  
       
      		 
      Error occurred on container - sequoiatools/couchbase-cli:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
      		 
       
      		 
      docker logs 9c1a88
      		 
      docker start 9c1a88
      		 
       
      		 
      8ERROR: Internal server error, please retry your request

      Restarting job will add more logs if see observe the same error.

      Attachments

        1. queries.png
          queries.png
          173 kB
        2. ns_server memory at OOM kill times.png
          ns_server memory at OOM kill times.png
          237 kB
        3. ns_server memory.png
          ns_server memory.png
          197 kB
        4. cpu on query node.png
          cpu on query node.png
          188 kB

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-28970 System Test : Query panics during indexer rebalance causing it to fail

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-29034 Query nodes activity impacted after failover of a data node due to network outage (net_tick_timeout)

          • Major - Major loss of function.
          • Closed
          For Gerrit Dashboard: MB-29028
          # Subject Branch Project Status CR V
          92416,3 MB-29028. Reduce Refresh() master go-couchbase Status: ABANDONED +2 0
          92417,2 MB-29028. Avoid ks.Count() on primary scan master query Status: MERGED +2 +1
          92534,2 MB-29028. Add GetCount() master go-couchbase Status: MERGED +2 +1

          Activity

            People

              vikas.chaudhary Vikas Chaudhary
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty