Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-28306

Service 'query' exited with status 134 - centos longevity

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.5.0
    • 5.5.0
    • query

    Description

      Following query service exit found in 5.5.0-1911 centos longevity:

      172.23.108.103 diag.log:

      2018-02-20T19:21:42.428-08:00, ns_log:0:info:message(ns_1@172.23.99.25) - Service 'query' exited with status 137. Restarting. Messages:
      2018-02-20T19:19:54.291-08:00 [INFO] Using plain authentication for user @cbq-engine 
      2018-02-20T19:19:54.291-08:00 [INFO] Using plain authentication for user @cbq-engine 
      2018-02-20T19:19:54.291-08:00 [INFO] Using plain authentication for user @cbq-engine 
      2018-02-20T19:19:54.291-08:00 [INFO] Using plain authentication for user @cbq-engine 
      2018-02-20T19:19:57.539-08:00 [Info] connected with 3 indexers
      2018-02-20T19:19:57.569-08:00 [Info] GSIC[default/default-1519163795502456181] logstats "default" {"gsi_scan_count":186240,"gsi_scan_duration":8226023493604998,"gsi_throttle_duration":5910862131355,"gsi_prime_duration":4605593043229069,"gsi_blocked_duration":717600393832501,"gsi_totalbackfills":177540}
      2018-02-20T19:19:58.121-08:00 [Info] GSIC[default/CUSTOMER-1519163860497638031] logstats "CUSTOMER" {"gsi_scan_count":97445,"gsi_scan_duration":11787660215481736,"gsi_throttle_duration":0,"gsi_prime_duration":298260055850031,"gsi_blocked_duration":0,"gsi_totalbackfills":0}
      [goport(/opt/couchbase/bin/cbq-engine)] 2018/02/20 19:21:39 child process exited with status 137
       
      -------------------------------
       
       
      per_node_processes('ns_1@172.23.108.103') =
           {<0.32765.73>,
            [{registered_name,'dcp_consumer_conn-WAREHOUSE-ns_1@172.23.108.104'},
             {status,waiting},
             {initial_call,{proc_lib,init_p,5}},
             {backtrace,[<<"Program counter: 0x00007f989f9c9920 (gen_server:loop/6 + 264)">>,
                         <<"CP: 0x0000000000000000 (invalid)">>,<<"arity = 0">>,
                         <<>>,
                         <<"0x00007f9851d4af68 Return addr 0x00007f989f8f0210 (proc_lib:init_p_do_apply/3 + 56)">>,
                         <<"y(0)     []">>,<<"y(1)     10000">>,
                         <<"y(2)     dcp_proxy">>,
                         <<"(3)     {state,#Port<0.124902>,{consumer,\"replication:ns_1@172.23.108.104->ns_1@172.23.108.103:WAREHOUSE\",'ns_1@172.23.1">>,
                         <<"y(4)     <0.32765.73>">>,<<"y(5)     <0.32696.73>">>,
                         <<>>,
                         <<"0x00007f9851d4afa0 Return addr 0x0000000000891848 (<terminate process normally>)">>,
                         <<"y(0)     Catch 0x00007f989f8f0230 (proc_lib:init_p_do_apply/3 + 88)">>,
                         <<>>]},
             {error_handler,error_handler},
             {garbage_collection,[{min_bin_vheap_size,46422},
                                  {min_heap_size,233},
                                  {fullsweep_after,512},
                                  {minor_gcs,152}]},
             {heap_size,1598},
             {total_heap_size,2585},
             {links,[<0.32696.73>,#Port<0.124902>]},
             {monitors,[]},
             {monitored_by,[<0.6289.0>]},
             {memory,21632},
             {messages,[]},
             {message_queue_len,0},
             {reductions,7882112},
             {trap_exit,false},
             {current_location,{gen_server,loop,6,
                                           [{file,"gen_server.erl"},{line,358}]}},
             {dictionary,[{'$ancestors',['dcp_replicator-WAREHOUSE-ns_1@172.23.108.104',
                                         'dcp_sup-WAREHOUSE',
                                         'single_bucket_kv_sup-WAREHOUSE',
                                         ns_bucket_sup,ns_bucket_worker_sup,
                                         ns_server_sup,ns_server_nodes_sup,
                                         <0.170.0>,ns_server_cluster_sup,<0.89.0>]},
                          {'$initial_call',{dcp_proxy,init,1}}]}]}
      

      172.23.99.25 - query log:

      2018-02-20T19:19:54.291-08:00 [INFO] Using plain authentication for user @cbq-engine 
      2018-02-20T19:19:54.291-08:00 [INFO] Using plain authentication for user @cbq-engine 
      2018-02-20T19:19:54.291-08:00 [INFO] Using plain authentication for user @cbq-engine 
      2018-02-20T19:19:54.291-08:00 [INFO] Using plain authentication for user @cbq-engine 
      2018-02-20T19:19:57.539-08:00 [Info] connected with 3 indexers
      2018-02-20T19:19:57.569-08:00 [Info] GSIC[default/default-1519163795502456181] logstats "default" {"gsi_scan_count":186240,"gsi_scan_duration":8226023493604998,"gsi_throttle_duration":5910862131355,"gsi_prime_duration":4605593043229069,"gsi_blocked_duration":717600393832501,"gsi_totalbackfills":177540}
      2018-02-20T19:19:58.121-08:00 [Info] GSIC[default/CUSTOMER-1519163860497638031] logstats "CUSTOMER" {"gsi_scan_count":97445,"gsi_scan_duration":11787660215481736,"gsi_throttle_duration":0,"gsi_prime_duration":298260055850031,"gsi_blocked_duration":0,"gsi_totalbackfills":0}
      [goport(/opt/couchbase/bin/cbq-engine)] 2018/02/20 19:21:39 child process exited with status 137
      _time=2018-02-20T19:21:58.173-08:00 _level=INFO _msg= Initialization of cbauth succeeded  
      _time=2018-02-20T19:21:59.303-08:00 _level=INFO _msg=New store created with url http://127.0.0.1:8091 
      _time=2018-02-20T19:21:59.303-08:00 _level=INFO _msg=pollEOF: About to start stdin polling 
      _time=2018-02-20T19:21:59.305-08:00 _level=INFO _msg=cbq-engine started servicers=32 plus-servicers=128 timeout=0s max-index-api=3 n1ql_feat_cntrl=0 version=2.0.0-N1QL datastore=http://127.0.0.1:8091 scan-cap=512 pipeline-cap=512 pipeline-batch=16 request-cap=256 max-concurrency=8 loglevel=INFO request-size-cap=67108864 
      _time=2018-02-20T19:21:59.307-08:00 _level=INFO _msg=HttpEndpoint: Listen Address=[::]:8093 
      _time=2018-02-20T19:21:59.307-08:00 _level=INFO _msg= Certificates have been refreshed by ns server  
      _time=2018-02-20T19:21:59.329-08:00 _level=INFO _msg=New settings received: {"query.settings.curl_whitelist":{"all_access":false,"allowed_urls":[],"disallowed_urls":[]},"query.settings.tmp_space_dir":"/opt/couchbase/var/lib/couchbase/tmp","query.settings.tmp_space_size":5120}
       
      _time=2018-02-20T19:21:59.336-08:00 _level=INFO _msg=HttpEndpoint: ListenTLS Address=[::]:18093 
      _time=2018-02-20T19:21:59.437-08:00 _level=INFO _msg=ERROR: Starting TLS listener - listen tcp :18093: bind: address already in use 
      _time=2018-02-20T19:21:59.483-08:00 _level=INFO _msg=Settings notifier from metakv
       
      _time=2018-02-20T19:21:59.483-08:00 _level=INFO _msg=New Value for curl whitelist <ud>map[disallowed_urls:[] all_access:false allowed_urls:[]]</ud> 
      _time=2018-02-20T19:21:59.503-08:00 _level=INFO _msg=Created New Bucket default 
      _time=2018-02-20T19:22:00.307-08:00 _level=INFO _msg=Created New Bucket CUSTOMER 
      2018-02-20T19:22:00.385-08:00 [Info] GSI client: removing old file /opt/couchbase/var/lib/couchbase/tmp/scan-results25882003408218 last-modified @ 2018-02-20 14:33:05.192534865 -0800 PST
      2018-02-20T19:22:00.387-08:00 [Info] New settings received: 
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty