Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29028

Query REST API retries cause ns_server memory to dramatically spike (was: Service 'ns_server' exited with status 137 observed during centos longevity test)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.5.0
    • 5.5.0
    • query
    • Centos cluster

    Description

      Build : 5.5.0-2361
      We observe Internal server error , after we failover 2 node in centos longevity test. 

      step

      [2018-03-31T09:56:05-07:00, sequoiatools/couchbase-cli:9d514c] server-add -c 172.23.108.103:8091 --server-add 172.23.108.104:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      		 
      [2018-03-31T09:58:43-07:00, sequoiatools/couchbase-cli:a2eaa1] failover -c 172.23.108.103:8091 --server-failover 172.23.96.56:8091 -u Administrator -p password
      		 
      [2018-03-31T10:19:43-07:00, sequoiatools/couchbase-cli:207967] failover -c 172.23.108.103:8091 --server-failover 172.23.97.238:8091 -u Administrator -p password --force
      		 
      [2018-03-31T10:20:00-07:00, sequoiatools/couchbase-cli:9c1a88] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      		 
      →  
       
      		 
      Error occurred on container - sequoiatools/couchbase-cli:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
      		 
       
      		 
      docker logs 9c1a88
      		 
      docker start 9c1a88
      		 
       
      		 
      8ERROR: Internal server error, please retry your request

      Restarting job will add more logs if see observe the same error.

      Attachments

        1. cpu on query node.png
          cpu on query node.png
          188 kB
        2. ns_server memory.png
          ns_server memory.png
          197 kB
        3. ns_server memory at OOM kill times.png
          ns_server memory at OOM kill times.png
          237 kB
        4. queries.png
          queries.png
          173 kB

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-28970 System Test : Query panics during indexer rebalance causing it to fail

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-29034 Query nodes activity impacted after failover of a data node due to network outage (net_tick_timeout)

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              vikas.chaudhary Vikas Chaudhary
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty