Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31697

Service goxdcr exited with status 137 - accompanied by projector exit and rebalance failure

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      ./testrunner -i /tmp/testexec.28803.ini -p stop-on-failure=False,get-coredumps=True,fail_on_errors=1,upgrade_version=6.5.0-1441,get-cbcollect-info=True -t xdcr.upgradeXDCR.UpgradeTests.online_cluster_upgrade,initial_version=4.6.3-4136,bucket_topology=default:1><2;standard_bucket_1:1<2;sasl_bucket_1:1><2,expires=500,GROUP=ONLINE

      Above test case fails with a rebalance failure in 6.5.0-1441 - there is a goxdcr service exit and a projector exit around the same timestamp:

       Rebalancing out all old version nodes
      [2018-10-18 01:19:13,159] - [rest_client:1462] INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.106.202,ns_1@172.23.106.204', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.106.204,ns_1@172.23.98.14,ns_1@172.23.106.202,ns_1@172.23.106.63'}
      [2018-10-18 01:19:13,167] - [rest_client:1467] INFO - rebalance operation started
      [2018-10-18 01:19:13,173] - [rest_client:1598] ERROR - {u'status': u'none', u'errorMessage': u'Please remove proxy ports from the following buckets: standard_bucket_1'} - rebalance failed
      [2018-10-18 01:19:13,203] - [rest_client:3135] INFO - Latest logs from UI on 172.23.106.202:
      [2018-10-18 01:19:13,203] - [rest_client:3136] ERROR - {u'node': u'ns_1@172.23.106.63', u'code': 0, u'text': u"Haven't heard from a higher priority node or a master, so I'm taking over.", u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:57.068Z', u'module': u'mb_master', u'tstamp': 1539850737068, u'type': u'info'}
      [2018-10-18 01:19:13,203] - [rest_client:3136] ERROR - {u'node': u'ns_1@172.23.98.14', u'code': 0, u'text': u'Service \'goxdcr\' exited with status 137. Restarting. Messages:\n2018-10-18T01:17:25.695-07:00 INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":1304816,"TotalAlloc":1304816,"Sys":5867768,"Lookups":18,"Mallocs":14630,"Frees":628,"HeapAlloc":1304816,"HeapSys":2490368,"HeapIdle":401408,"HeapInuse":2088960,"HeapReleased":0,"HeapObjects":14002,"StackInuse":655360,"StackSys":655360,"MSpanInuse":35720,"MSpanSys":49152,"MCacheInuse":4800,"MCacheSys":16384,"BuckHashSys":1443357,"GCSys":169984,"OtherSys":1043163,"NextGC":4473924,"LastGC":0,"PauseTotalNs":0,"PauseNs":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"PauseEnd":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"NumGC":0,"NumForcedGC":0,"GCCPUFraction":0,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":228,"Frees":0},{"Size":16,"Mallocs":3158,"Frees":0},{"Size":32,"Mallocs":5328,"Frees":0},{"Size":48,"Mallocs":1744,"Frees":0},{"Size":64,"Mallocs":655,"Frees":0},{"Size":80,"Mallocs":439,"Frees":0},{"Size":96,"Mallocs":558,"Frees":0},{"Size":112,"Mallocs":488,"Frees":0},{"Size":128,"Mallocs":151,"Frees":0},{"Size":144,"Mallocs":68,"Frees":0},{"Size":160,"Mallocs":73,"Frees":0},{"Size":176,"Mallocs":116,"Frees":0},{"Size":192,"Mallocs":8,"Frees":0},{"Size":208,"Mallocs":134,"Frees":0},{"Size":224,"Mallocs":75,"Frees":0},{"Size":240,"Mallocs":2,"Frees":0},{"Size":256,"Mallocs":117,"Frees":0},{"Size":288,"Mallocs":50,"Frees":0},{"Size":320,"Mallocs":18,"Frees":0},{"Size":352,"Mallocs":270,"Frees":0},{"Size":384,"Mallocs":3,"Frees":0},{"Size":416,"Mallocs":80,"Frees":0},{"Size":448,"Mallocs":9,"Frees":0},{"Size":480,"Mallocs":4,"Frees":0},{"Size":512,"Mallocs":15,"Frees":0},{"Size":576,"Mallocs":17,"Frees":0},{"Size":640,"Mallocs":21,"Frees":0},{"Size":704,"Mallocs":25,"Frees":0},{"Size":768,"Mallocs":2,"Frees":0},{"Size":896,"Mallocs":12,"Frees":0},{"Size":1024,"Mallocs":13,"Frees":0},{"Size":1152,"Mallocs":15,"Frees":0},{"Size":1280,"Mallocs":12,"Frees":0},{"Size":1408,"Mallocs":1,"Frees":0},{"Size":1536,"Mallocs":2,"Frees":0},{"Size":1792,"Mallocs":7,"Frees":0},{"Size":2048,"Mallocs":22,"Frees":0},{"Size":2304,"Mallocs":9,"Frees":0},{"Size":2688,"Mallocs":7,"Frees":0},{"Size":3072,"Mallocs":2,"Frees":0},{"Size":3200,"Mallocs":1,"Frees":0},{"Size":3456,"Mallocs":5,"Frees":0},{"Size":4096,"Mallocs":20,"Frees":0},{"Size":4864,"Mallocs":8,"Frees":0},{"Size":5376,"Mallocs":1,"Frees":0},{"Size":6144,"Mallocs":4,"Frees":0},{"Size":6528,"Mallocs":0,"Frees":0},{"Size":6784,"Mallocs":1,"Frees":0},{"Size":6912,"Mallocs":0,"Frees":0},{"Size":8192,"Mallocs":2,"Frees":0},{"Size":9472,"Mallocs":0,"Frees":0},{"Size":9728,"Mallocs":0,"Frees":0},{"Size":10240,"Mallocs":0,"Frees":0},{"Size":10880,"Mallocs":0,"Frees":0},{"Size":12288,"Mallocs":0,"Frees":0},{"Size":13568,"Mallocs":0,"Frees":0},{"Size":14336,"Mallocs":0,"Frees":0},{"Size":16384,"Mallocs":0,"Frees":0},{"Size":18432,"Mallocs":0,"Frees":0},{"Size":19072,"Mallocs":0,"Frees":0}]}\n', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:54.097Z', u'module': u'ns_log', u'tstamp': 1539850734097, u'type': u'info'}
      [2018-10-18 01:19:13,203] - [rest_client:3136] ERROR - {u'node': u'ns_1@172.23.98.14', u'code': 0, u'text': u'Service \'projector\' exited with status 137. Restarting. Messages:\n2018-10-18T01:16:43.885-07:00 [Info] memstats {"Alloc":1017760, "TotalAlloc":1521320, "Sys":4820992, "Lookups":11, "Mallocs":8372,"Frees":1544, "HeapAlloc":1017760, "HeapSys":2555904, "HeapIdle":712704, "HeapInuse":1843200,"HeapReleased":0, "HeapObjects":6828,"GCSys":313344, "LastGC":1539850543884852734,"PauseTotalNs":1350180, "PauseNs":[663606, 686574], "NumGC":2}\n2018-10-18T01:17:43.885-07:00 [Info] memstats {"Alloc":1019696, "TotalAlloc":1523256, "Sys":4820992, "Lookups":11, "Mallocs":8406,"Frees":1553, "HeapAlloc":1019696, "HeapSys":2555904, "HeapIdle":712704, "HeapInuse":1843200,"HeapReleased":0, "HeapObjects":6853,"GCSys":313344, "LastGC":1539850543884852734,"PauseTotalNs":1350180, "PauseNs":[], "NumGC":2}\n2018-10-18T01:18:43.885-07:00 [Info] memstats {"Alloc":829360, "TotalAlloc":1527192, "Sys":4820992, "Lookups":11, "Mallocs":8445,"Frees":2281, "HeapAlloc":829360, "HeapSys":2555904, "HeapIdle":974848, "HeapInuse":1581056,"HeapReleased":0, "HeapObjects":6164,"GCSys":313344, "LastGC":1539850663886345869,"PauseTotalNs":1500420, "PauseNs":[150240], "NumGC":3}\n', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:54.096Z', u'module': u'ns_log', u'tstamp': 1539850734096, u'type': u'info'}
      [2018-10-18 01:19:13,203] - [rest_client:3136] ERROR - {u'node': u'ns_1@172.23.106.63', u'code': 0, u'text': u'Service \'goxdcr\' exited with status 137. Restarting. Messages:\n2018-10-18T01:17:40.231-07:00 INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":1309936,"TotalAlloc":1309936,"Sys":4427776,"Lookups":18,"Mallocs":14653,"Frees":628,"HeapAlloc":1309936,"HeapSys":2523136,"HeapIdle":409600,"HeapInuse":2113536,"HeapReleased":0,"HeapObjects":14025,"StackInuse":622592,"StackSys":622592,"MSpanInuse":36632,"MSpanSys":49152,"MCacheInuse":4800,"MCacheSys":16384,"BuckHashSys":2917,"GCSys":169984,"OtherSys":1043611,"NextGC":4473924,"LastGC":0,"PauseTotalNs":0,"PauseNs":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"PauseEnd":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"NumGC":0,"NumForcedGC":0,"GCCPUFraction":0,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":228,"Frees":0},{"Size":16,"Mallocs":3160,"Frees":0},{"Size":32,"Mallocs":5329,"Frees":0},{"Size":48,"Mallocs":1745,"Frees":0},{"Size":64,"Mallocs":659,"Frees":0},{"Size":80,"Mallocs":450,"Frees":0},{"Size":96,"Mallocs":563,"Frees":0},{"Size":112,"Mallocs":488,"Frees":0},{"Size":128,"Mallocs":151,"Frees":0},{"Size":144,"Mallocs":68,"Frees":0},{"Size":160,"Mallocs":72,"Frees":0},{"Size":176,"Mallocs":111,"Frees":0},{"Size":192,"Mallocs":8,"Frees":0},{"Size":208,"Mallocs":135,"Frees":0},{"Size":224,"Mallocs":75,"Frees":0},{"Size":240,"Mallocs":2,"Frees":0},{"Size":256,"Mallocs":117,"Frees":0},{"Size":288,"Mallocs":49,"Frees":0},{"Size":320,"Mallocs":18,"Frees":0},{"Size":352,"Mallocs":266,"Frees":0},{"Size":384,"Mallocs":4,"Frees":0},{"Size":416,"Mallocs":85,"Frees":0},{"Size":448,"Mallocs":11,"Frees":0},{"Size":480,"Mallocs":4,"Frees":0},{"Size":512,"Mallocs":14,"Frees":0},{"Size":576,"Mallocs":17,"Frees":0},{"Size":640,"Mallocs":21,"Frees":0},{"Size":704,"Mallocs":25,"Frees":0},{"Size":768,"Mallocs":3,"Frees":0},{"Size":896,"Mallocs":12,"Frees":0},{"Size":1024,"Mallocs":12,"Frees":0},{"Size":1152,"Mallocs":15,"Frees":0},{"Size":1280,"Mallocs":12,"Frees":0},{"Size":1408,"Mallocs":1,"Frees":0},{"Size":1536,"Mallocs":3,"Frees":0},{"Size":1792,"Mallocs":8,"Frees":0},{"Size":2048,"Mallocs":22,"Frees":0},{"Size":2304,"Mallocs":9,"Frees":0},{"Size":2688,"Mallocs":7,"Frees":0},{"Size":3072,"Mallocs":2,"Frees":0},{"Size":3200,"Mallocs":1,"Frees":0},{"Size":3456,"Mallocs":5,"Frees":0},{"Size":4096,"Mallocs":20,"Frees":0},{"Size":4864,"Mallocs":8,"Frees":0},{"Size":5376,"Mallocs":1,"Frees":0},{"Size":6144,"Mallocs":4,"Frees":0},{"Size":6528,"Mallocs":0,"Frees":0},{"Size":6784,"Mallocs":1,"Frees":0},{"Size":6912,"Mallocs":0,"Frees":0},{"Size":8192,"Mallocs":2,"Frees":0},{"Size":9472,"Mallocs":0,"Frees":0},{"Size":9728,"Mallocs":0,"Frees":0},{"Size":10240,"Mallocs":0,"Frees":0},{"Size":10880,"Mallocs":0,"Frees":0},{"Size":12288,"Mallocs":0,"Frees":0},{"Size":13568,"Mallocs":0,"Frees":0},{"Size":14336,"Mallocs":0,"Frees":0},{"Size":16384,"Mallocs":0,"Frees":0},{"Size":18432,"Mallocs":0,"Frees":0},{"Size":19072,"Mallocs":0,"Frees":0}]}\n', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:51.557Z', u'module': u'ns_log', u'tstamp': 1539850731557, u'type': u'info'}
      [2018-10-18 01:19:13,203] - [rest_client:3136] ERROR - {u'node': u'ns_1@172.23.106.63', u'code': 0, u'text': u'Service \'projector\' exited with status 137. Restarting. Messages:\n2018-10-18T01:15:46.944-07:00 [Info] PROJ[:9999] settings indexer.settings.inmemory_snapshot.interval will updated to `200`\n2018-10-18T01:15:46.944-07:00 [Info] PROJ[:9999] settings indexer.settings.moi.persistence_threads will updated to `2`\n2018-10-18T01:15:46.944-07:00 [Info] amd64 linux; cpus: 4; GOMAXPROCS: 4; version: go1.7.6\n2018-10-18T01:16:46.941-07:00 [Info] memstats {"Alloc":1024208, "TotalAlloc":1527640, "Sys":6260984, "Lookups":11, "Mallocs":8390,"Frees":1560, "HeapAlloc":1024208, "HeapSys":2621440, "HeapIdle":745472, "HeapInuse":1875968,"HeapReleased":0, "HeapObjects":6830,"GCSys":311296, "LastGC":1539850546940910871,"PauseTotalNs":2026079, "PauseNs":[1185023, 841056], "NumGC":2}\n2018-10-18T01:17:46.941-07:00 [Info] memstats {"Alloc":1026352, "TotalAlloc":1529784, "Sys":6260984, "Lookups":11, "Mallocs":8424,"Frees":1569, "HeapAlloc":1026352, "HeapSys":2621440, "HeapIdle":745472, "HeapInuse":1875968,"HeapReleased":0, "HeapObjects":6855,"GCSys":311296, "LastGC":1539850546940910871,"PauseTotalNs":2026079, "PauseNs":[], "NumGC":2}\n', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:51.555Z', u'module': u'ns_log', u'tstamp': 1539850731555, u'type': u'info'}
      [2018-10-18 01:19:13,204] - [rest_client:3136] ERROR - {u'node': u'ns_1@172.23.98.14', u'code': 0, u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"client_cert_auth">>,\n                                                                      <<"collections_enabled">>,\n                                                                      <<"datatype_snappy">>,\n                                                                      <<"xattr_enabled">>]', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:48.904Z', u'module': u'memcached_config_mgr', u'tstamp': 1539850728904, u'type': u'info'}
      [2018-10-18 01:19:13,204] - [rest_client:3136] ERROR - {u'node': u'ns_1@172.23.98.14', u'code': 3, u'text': u'Node ns_1@172.23.98.14 joined cluster', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:48.830Z', u'module': u'ns_cluster', u'tstamp': 1539850728830, u'type': u'info'}
      [2018-10-18 01:19:13,204] - [rest_client:3136] ERROR - {u'node': u'ns_1@172.23.98.14', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.98.14\'. Version: "6.5.0-1441-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2018-10-18T01:18:48.778Z', u'module': u'menelaus_sup', u'tstamp': 1539850728778, u'type': u'info'}
      [2018-10-18 01:19:13,204] - [rest_client:3136] ERROR - {u'node': u'ns_1@172.23.106.202', u'code': 0, u'text': u'Conflicting configuration changes to field cbas_memory_quota:\n1166 and\n1185, choosing the former.', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:48.772Z', u'module': u'ns_config', u'tstamp': 1539850728772, u'type': u'info'}
      [2018-10-18 01:19:13,204] - [rest_client:3136] ERROR - {u'node': u'ns_1@172.23.106.204', u'code': 0, u'text': u'Conflicting configuration changes to field cbas_memory_quota:\n1166 and\n1185, choosing the former.', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:48.771Z', u'module': u'ns_config', u'tstamp': 1539850728771, u'type': u'info'}
      ERROR
      [('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 77, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 524, 'check', 'self.set_exception(ex)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
      Thu Oct 18 01:19:13 2018
      [('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('testrunner.py', 297, 'run', '**self._Thread__kwargs)'), ('/usr/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib/python2.7/unittest/case.py', 391, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.7/unittest/case.py', 327, 'run', 'testMethod()'), ('pytests/xdcr/upgradeXDCR.py', 406, 'online_cluster_upgrade', 'self._online_upgrade(self.src_nodes, self.servers[self.src_init + self.dest_init:])'), ('pytests/xdcr/upgradeXDCR.py', 164, '_online_upgrade', 'self.cluster.rebalance(update_servers + extra_servers, [], update_servers)'), ('lib/couchbase_helper/cluster.py', 333, 'rebalance', 'return _task.result(timeout)'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')]
      [2018-10-18 01:19:13,226] - [xdcrnewbasetests:2617] INFO - Collecting logs @ 172.23.106.202
      grabbing cbcollect from 172.23.106.202
      [2018-10-18 01:19:13,231] - [remote_util:233] INFO - connecting to 172.23.106.202 with username:root pas
      

      From diag.log:

      2018-10-18T01:18:48.904-07:00, memcached_config_mgr:0:info:message(ns_1@172.23.98.14) - Hot-reloaded memcached.json for config change of the following keys: [<<"client_cert_auth">>,
                                                                            <<"collections_enabled">>,
                                                                            <<"datatype_snappy">>,
                                                                            <<"xattr_enabled">>]
      2018-10-18T01:18:51.555-07:00, ns_log:0:info:message(ns_1@172.23.106.63) - Service 'projector' exited with status 137. Restarting. Messages:
      2018-10-18T01:15:46.944-07:00 [Info] PROJ[:9999] settings indexer.settings.inmemory_snapshot.interval will updated to `200`
      2018-10-18T01:15:46.944-07:00 [Info] PROJ[:9999] settings indexer.settings.moi.persistence_threads will updated to `2`
      2018-10-18T01:15:46.944-07:00 [Info] amd64 linux; cpus: 4; GOMAXPROCS: 4; version: go1.7.6
      2018-10-18T01:16:46.941-07:00 [Info] memstats {"Alloc":1024208, "TotalAlloc":1527640, "Sys":6260984, "Lookups":11, "Mallocs":8390,"Frees":1560, "HeapAlloc":1024208, "HeapSys":2621440, "HeapIdle":745472, "HeapInuse":1875968,"HeapReleased":0, "HeapObjects":6830,"GCSys":311296, "LastGC":1539850546940910871,"PauseTotalNs":2026079, "PauseNs":[1185023, 841056], "NumGC":2}
      2018-10-18T01:17:46.941-07:00 [Info] memstats {"Alloc":1026352, "TotalAlloc":1529784, "Sys":6260984, "Lookups":11, "Mallocs":8424,"Frees":1569, "HeapAlloc":1026352, "HeapSys":2621440, "HeapIdle":745472, "HeapInuse":1875968,"HeapReleased":0, "HeapObjects":6855,"GCSys":311296, "LastGC":1539850546940910871,"PauseTotalNs":2026079, "PauseNs":[], "NumGC":2}
       
      2018-10-18T01:18:51.557-07:00, ns_log:0:info:message(ns_1@172.23.106.63) - Service 'goxdcr' exited with status 137. Restarting. Messages:
      2018-10-18T01:17:40.231-07:00 INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":1309936,"TotalAlloc":1309936,"Sys":4427776,"Lookups":18,"Mallocs":14653,"Frees":628,"HeapAlloc":1309936,"HeapSys":2523136,"HeapIdle":409600,"HeapInuse":2113536,"HeapReleased":0,"HeapObjects":14025,"StackInuse":622592,"StackSys":622592,"MSpanInuse":36632,"MSpanSys":49152,"MCacheInuse":4800,"MCacheSys":16384,"BuckHashSys":2917,"GCSys":169984,"OtherSys":1043611,"NextGC":4473924,"LastGC":0,"PauseTotalNs":0,"PauseNs":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"PauseEnd":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"NumGC":0,"NumForcedGC":0,"GCCPUFraction":0,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":228,"Frees":0},{"Size":16,"Mallocs":3160,"Frees":0},{"Size":32,"Mallocs":5329,"Frees":0},{"Size":48,"Mallocs":1745,"Frees":0},{"Size":64,"Mallocs":659,"Frees":0},{"Size":80,"Mallocs":450,"Frees":0},{"Size":96,"Mallocs":563,"Frees":0},{"Size":112,"Mallocs":488,"Frees":0},{"Size":128,"Mallocs":151,"Frees":0},{"Size":144,"Mallocs":68,"Frees":0},{"Size":160,"Mallocs":72,"Frees":0},{"Size":176,"Mallocs":111,"Frees":0},{"Size":192,"Mallocs":8,"Frees":0},{"Size":208,"Mallocs":135,"Frees":0},{"Size":224,"Mallocs":75,"Frees":0},{"Size":240,"Mallocs":2,"Frees":0},{"Size":256,"Mallocs":117,"Frees":0},{"Size":288,"Mallocs":49,"Frees":0},{"Size":320,"Mallocs":18,"Frees":0},{"Size":352,"Mallocs":266,"Frees":0},{"Size":384,"Mallocs":4,"Frees":0},{"Size":416,"Mallocs":85,"Frees":0},{"Size":448,"Mallocs":11,"Frees":0},{"Size":480,"Mallocs":4,"Frees":0},{"Size":512,"Mallocs":14,"Frees":0},{"Size":576,"Mallocs":17,"Frees":0},{"Size":640,"Mallocs":21,"Frees":0},{"Size":704,"Mallocs":25,"Frees":0},{"Size":768,"Mallocs":3,"Frees":0},{"Size":896,"Mallocs":12,"Frees":0},{"Size":1024,"Mallocs":12,"Frees":0},{"Size":1152,"Mallocs":15,"Frees":0},{"Size":1280,"Mallocs":12,"Frees":0},{"Size":1408,"Mallocs":1,"Frees":0},{"Size":1536,"Mallocs":3,"Frees":0},{"Size":1792,"Mallocs":8,"Frees":0},{"Size":2048,"Mallocs":22,"Frees":0},{"Size":2304,"Mallocs":9,"Frees":0},{"Size":2688,"Mallocs":7,"Frees":0},{"Size":3072,"Mallocs":2,"Frees":0},{"Size":3200,"Mallocs":1,"Frees":0},{"Size":3456,"Mallocs":5,"Frees":0},{"Size":4096,"Mallocs":20,"Frees":0},{"Size":4864,"Mallocs":8,"Frees":0},{"Size":5376,"Mallocs":1,"Frees":0},{"Size":6144,"Mallocs":4,"Frees":0},{"Size":6528,"Mallocs":0,"Frees":0},{"Size":6784,"Mallocs":1,"Frees":0},{"Size":6912,"Mallocs":0,"Frees":0},{"Size":8192,"Mallocs":2,"Frees":0},{"Size":9472,"Mallocs":0,"Frees":0},{"Size":9728,"Mallocs":0,"Frees":0},{"Size":10240,"Mallocs":0,"Frees":0},{"Size":10880,"Mallocs":0,"Frees":0},{"Size":12288,"Mallocs":0,"Frees":0},{"Size":13568,"Mallocs":0,"Frees":0},{"Size":14336,"Mallocs":0,"Frees":0},{"Size":16384,"Mallocs":0,"Frees":0},{"Size":18432,"Mallocs":0,"Frees":0},{"Size":19072,"Mallocs":0,"Frees":0}]}
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            pavithra.mahamani Pavithra Mahamani (Inactive)
            arunkumar Arunkumar Senthilnathan (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty