[System test] Rebalancing a node out or failing over a node in longevity cluster causes service crashes

2018-10-30T23:42:28.050-07:00, memcached_config_mgr:0:info:message(ns_1@172.23.97.242) - Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]

2018-10-30T23:42:43.138-07:00, ns_orchestrator:0:info:message(ns_1@172.23.108.103) - Starting rebalance, KeepNodes = ['ns_1@172.23.104.164','ns_1@172.23.104.61',

                                 'ns_1@172.23.104.67','ns_1@172.23.104.69',

                                 'ns_1@172.23.104.70','ns_1@172.23.104.87',

                                 'ns_1@172.23.104.88','ns_1@172.23.106.188',

                                 'ns_1@172.23.108.103','ns_1@172.23.108.104',

                                 'ns_1@172.23.96.145','ns_1@172.23.96.148',

                                 'ns_1@172.23.96.168','ns_1@172.23.96.253',

                                 'ns_1@172.23.96.56','ns_1@172.23.96.95',

                                 'ns_1@172.23.97.238','ns_1@172.23.97.239',

                                 'ns_1@172.23.97.242','ns_1@172.23.98.135',

                                 'ns_1@172.23.99.20','ns_1@172.23.99.21',

                                 'ns_1@172.23.99.25'], EjectNodes = ['ns_1@172.23.99.11'], Failed over and being ejected nodes = []; no delta recovery nodes

2018-10-30T23:42:47.358-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket WAREHOUSE

2018-10-30T23:42:47.809-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "WAREHOUSE" rebalance appears to be swap rebalance

2018-10-30T23:42:48.130-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket STOCK

2018-10-30T23:42:48.710-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "STOCK" rebalance appears to be swap rebalance

2018-10-30T23:42:48.928-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket ORDER_LINE

2018-10-30T23:42:49.414-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "ORDER_LINE" rebalance appears to be swap rebalance

2018-10-30T23:42:49.948-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket ORDERS

2018-10-30T23:42:50.550-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "ORDERS" rebalance appears to be swap rebalance

2018-10-30T23:42:50.845-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket NEW_ORDER

2018-10-30T23:42:51.374-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "NEW_ORDER" rebalance appears to be swap rebalance

2018-10-30T23:42:51.814-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket ITEM

2018-10-30T23:42:52.188-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "ITEM" rebalance appears to be swap rebalance

2018-10-30T23:42:52.507-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket HISTORY

2018-10-30T23:42:52.969-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "HISTORY" rebalance appears to be swap rebalance

2018-10-30T23:42:53.244-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket DISTRICT

2018-10-30T23:42:53.814-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "DISTRICT" rebalance appears to be swap rebalance

2018-10-30T23:42:53.884-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket CUSTOMER

2018-10-30T23:42:54.254-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "CUSTOMER" rebalance appears to be swap rebalance

2018-10-30T23:42:54.337-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket default

2018-10-30T23:42:55.010-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "default" rebalance appears to be swap rebalance

2018-10-30T23:49:05.458-07:00, ns_cluster:1:info:message(ns_1@172.23.99.11) - Node 'ns_1@172.23.99.11' is leaving cluster.

2018-10-30T23:49:05.660-07:00, ns_orchestrator:0:info:message(ns_1@172.23.108.103) - Rebalance completed successfully.

2018-10-30T23:49:07.936-07:00, ns_log:0:info:message(ns_1@172.23.99.11) - Service 'eventing' exited with status 137. Restarting. Messages:

2018-10-30T23:49:05.388-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.394-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.395-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.396-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.403-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.405-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.409-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.412-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.414-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.427-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.432-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.433-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.448-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.449-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.457-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.462-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.467-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:05.471-07:00 [Info] [gocb] Threshold Log:

2018-10-30T23:49:07.938-07:00, ns_log:0:info:message(ns_1@172.23.99.11) - Service 'goxdcr' exited with status 137. Restarting. Messages:

2018-10-30T23:47:28.819-07:00 INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":859088,"TotalAlloc":28052288,"Sys":9509112,"Lookups":18,"Mallocs":418106,"Frees":410808,"HeapAlloc":859088,"HeapSys":5406720,"HeapIdle":3252224,"HeapInuse":2154496,"HeapReleased":2998272,"HeapObjects":7298,"StackInuse":884736,"StackSys":884736,"MSpanInuse":44536,"MSpanSys":98304,"MCacheInuse":4800,"MCacheSys":16384,"BuckHashSys":1448285,"GCSys":438272,"OtherSys":1216411,"NextGC":4194304,"LastGC":1540968447905049851,"PauseTotalNs":33123345,"PauseNs":[165715,217788,1127022,712245,68660,123566,91793,1233567,1458447,128565,179979,579222,107289,76684,159442,71569,305632,143264,1683962,1032613,172160,124857,173578,609789,154532,123271,109585,85158,99827,109256,182610,168574,82316,93903,74917,89093,567148,179511,126450,209726,114709,265878,140692,152460,91398,256839,90184,80987,74589,637699,199421,115388,157720,210269,585895,87239,130856,14779651,279764,414964,85739,418184,66374,74242,108869,119354,101990,78706,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"PauseEnd":[1540960402318687287,1540960522819480604,1540960643321084726,1540960763730897320,1540960883802168657,1540961003820349235,1540961124404037094,1540961244464060181,1540961364528577266,1540961484586298831,1540961604643278991,1540961724710003386,1540961844764079933,1540961964820118805,1540962084887244007,1540962204943729566,1540962325006507742,1540962445068505123,1540962565168819735,1540962685227282794,1540962805281473844,1540962925355352835,1540963045412918729,1540963165487736156,1540963285552639225,1540963405612636138,1540963525743471538,1540963645807211111,1540963765819548604,1540963885916321778,1540964005968218476,1540964126013231133,1540964246060120900,1540964366105325200,1540964486164251410,1540964606212244246,1540964726261562027,1540964846309580217,1540964966319553983,1540965086417266884,1540965206480194284,1540965326532448028,1540965446581232571,1540965566633336631,1540965686684554047,1540965806751493946,1540965926801443784,1540966046819631488,1540966166906145914,1540966286954973575,1540966407012649182,1540966527061698405,1540966647114357102,1540966767162320905,1540966887228795896,1540967007277591812,1540967127324753229,1540967247404357892,1540967367449977928,1540967487497617918,1540967607557064581,1540967727605530969,1540967847654142394,1540967967702402725,1540968087750463087,1540968207802278275,1540968327848181500,1540968447905049851,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"NumGC":68,"NumForcedGC":0,"GCCPUFraction":0.000005242823593443896,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":5802,"Frees":5752},{"Size":16,"Mallocs":139816,"Frees":139010},{"Size":32,"Mallocs":60218,"Frees":55867},{"Size":48,"Mallocs":80138,"Frees":79670},{"Size":64,"Mallocs":22938,"Frees":22649},{"Size":80,"Mallocs":6519,"Frees":6045},{"Size":96,"Mallocs":22903,"Frees":22766},{"Size":112,"Mallocs":5808,"Frees":5694},{"Size":128,"Mallocs":5413,"Frees":5390},{"Size":144,"Mallocs":182,"Frees":131},{"Size":160,"Mallocs":111,"Frees":70},{"Size":176,"Mallocs":5476,"Frees":5444},{"Size":192,"Mallocs":76,"Frees":74},{"Size":208,"Mallocs":5978,"Frees":5947},{"Size":224,"Mallocs":5296,"Frees":5291},{"Size":240,"Mallocs":2,"Frees":0},{"Size":256,"Mallocs":5361,"Frees":5336},{"Size":288,"Mallocs":159,"Frees":141},{"Size":320,"Mallocs":55,"Frees":51},{"Size":352,"Mallocs":16013,"Frees":15994},{"Size":384,"Mallocs":4,"Frees":1},{"Size":416,"Mallocs":207,"Frees":8},{"Size":448,"Mallocs":9,"Frees":4},{"Size":480,"Mallocs":41,"Frees":38},{"Si

2018-10-31T01:14:09.662-07:00, analytics:0:info:message(ns_1@172.23.106.188) - Disconnected link "Default.Local".

2018-10-31T01:15:34.224-07:00, ns_orchestrator:0:info:message(ns_1@172.23.108.103) - Starting rebalance, KeepNodes = ['ns_1@172.23.104.164','ns_1@172.23.104.61',

                                 'ns_1@172.23.104.67','ns_1@172.23.104.69',

                                 'ns_1@172.23.104.70','ns_1@172.23.104.87',

                                 'ns_1@172.23.104.88','ns_1@172.23.108.103',

                                 'ns_1@172.23.108.104','ns_1@172.23.96.145',

                                 'ns_1@172.23.96.148','ns_1@172.23.96.168',

                                 'ns_1@172.23.96.253','ns_1@172.23.96.56',

                                 'ns_1@172.23.96.95','ns_1@172.23.97.238',

                                 'ns_1@172.23.97.239','ns_1@172.23.97.242',

                                 'ns_1@172.23.98.135','ns_1@172.23.99.11',

                                 'ns_1@172.23.99.20','ns_1@172.23.99.21',

                                 'ns_1@172.23.99.25'], EjectNodes = ['ns_1@172.23.106.188'], Failed over and being ejected nodes = []; no delta recovery nodes

2018-10-31T01:15:37.838-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket WAREHOUSE

2018-10-31T01:15:38.260-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "WAREHOUSE" rebalance appears to be swap rebalance

2018-10-31T01:15:38.600-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket STOCK

2018-10-31T01:15:39.163-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "STOCK" rebalance appears to be swap rebalance

2018-10-31T01:15:39.300-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket ORDER_LINE

2018-10-31T01:15:39.931-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "ORDER_LINE" rebalance appears to be swap rebalance

2018-10-31T01:15:40.047-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket ORDERS

2018-10-31T01:15:40.504-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "ORDERS" rebalance appears to be swap rebalance

2018-10-31T01:15:40.789-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket NEW_ORDER

2018-10-31T01:15:41.152-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "NEW_ORDER" rebalance appears to be swap rebalance

2018-10-31T01:15:41.512-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket ITEM

2018-10-31T01:15:41.840-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "ITEM" rebalance appears to be swap rebalance

2018-10-31T01:15:41.949-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket HISTORY

2018-10-31T01:15:42.375-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "HISTORY" rebalance appears to be swap rebalance

2018-10-31T01:15:42.555-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket DISTRICT

2018-10-31T01:15:42.951-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "DISTRICT" rebalance appears to be swap rebalance

2018-10-31T01:15:43.123-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket CUSTOMER

2018-10-31T01:15:43.475-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "CUSTOMER" rebalance appears to be swap rebalance

2018-10-31T01:15:43.737-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket default

2018-10-31T01:15:44.132-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "default" rebalance appears to be swap rebalance

2018-10-31T01:23:51.839-07:00, ns_log:0:info:message(ns_1@172.23.106.188) - Service 'cbas' exited with status 1. Restarting. Messages:

2018-10-31T01:23:48.028-07:00 INFO CBAS.cbas running /opt/couchbase/lib/cbas/runtime/bin/java args:[-XX:+ExitOnOutOfMemoryError -XX:MaxGCPauseMillis=60000 -Xmx20777m -Dfile.encoding=UTF-8 -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -DlogDir=/opt/couchbase/var/lib/couchbase/logs -cp /opt/couchbase/lib/cbas/repo/algebricks-common.jar:/opt/couchbase/lib/cbas/repo/algebricks-compiler.jar:/opt/couchbase/lib/cbas/repo/algebricks-core.jar:/opt/couchbase/lib/cbas/repo/algebricks-data.jar:/opt/couchbase/lib/cbas/repo/algebricks-rewriter.jar:/opt/couchbase/lib/cbas/repo/algebricks-runtime.jar:/opt/couchbase/lib/cbas/repo/args4j-2.33.jar:/opt/couchbase/lib/cbas/repo/asterix-active.jar:/opt/couchbase/lib/cbas/repo/asterix-algebra.jar:/opt/couchbase/lib/cbas/repo/asterix-app.jar:/opt/couchbase/lib/cbas/repo/asterix-common.jar:/opt/couchbase/lib/cbas/repo/asterix-external-data.jar:/opt/couchbase/lib/cbas/repo/asterix-lang-aql.jar:/opt/couchbase/lib/cbas/repo/asterix-lang-common.jar:/opt/couchbase/lib/cbas/repo/asterix-lang-sqlpp.jar:/opt/couchbase/lib/cbas/repo/asterix-metadata.jar:/opt/couchbase/lib/cbas/repo/asterix-om.jar:/opt/couchbase/lib/cbas/repo/asterix-replication.jar:/opt/couchbase/lib/cbas/repo/asterix-runtime.jar:/opt/couchbase/lib/cbas/repo/asterix-transactions.jar:/opt/couchbase/lib/cbas/repo/cbas-common.jar:/opt/couchbase/lib/cbas/repo/cbas-connector.jar:/opt/couchbase/lib/cbas/repo/cbas-dcp-client.jar:/opt/couchbase/lib/cbas/repo/cbas-server.jar:/opt/couchbase/lib/cbas/repo/commons-codec-1.9.jar:/opt/couchbase/lib/cbas/repo/commons-collections4-4.1.jar:/opt/couchbase/lib/cbas/repo/commons-compress-1.12.jar:/opt/couchbase/lib/cbas/repo/commons-io-2.6.jar:/opt/couchbase/lib/cbas/repo/commons-lang3-3.7.jar:/opt/couchbase/lib/cbas/repo/commons-logging-1.2.jar:/opt/couchbase/lib/cbas/repo/commons-text-1.2.jar:/opt/couchbase/lib/cbas/repo/core-io-1.6.1.jar:/opt/couchbase/lib/cbas/repo/disruptor-1.2.13-jdk8.jar:/opt/couchbase/lib/cbas/repo/esri-geometry-api-2.0.0.jar:/opt/couchbase/lib/cbas/repo/fastutil-8.1.1.jar:/opt/couchbase/lib/cbas/repo/guava-18.0.jar:/opt/couchbase/lib/cbas/repo/httpclient-4.5.2.jar:/opt/couchbase/lib/cbas/repo/httpcore-4.4.5.jar:/opt/couchbase/lib/cbas/repo/hyracks-api.jar:/opt/couchbase/lib/cbas/repo/hyracks-client.jar:/opt/couchbase/lib/cbas/repo/hyracks-comm.jar:/opt/couchbase/lib/cbas/repo/hyracks-control-cc.jar:/opt/couchbase/lib/cbas/repo/hyracks-control-common.jar:/opt/couchbase/lib/cbas/repo/hyracks-control-nc.jar:/opt/couchbase/lib/cbas/repo/hyracks-data-std.jar:/opt/couchbase/lib/cbas/repo/hyracks-dataflow-common.jar:/opt/couchbase/lib/cbas/repo/hyracks-dataflow-std.jar:/opt/couchbase/lib/cbas/repo/hyracks-http.jar:/opt/couchbase/lib/cbas/repo/hyracks-ipc.jar:/opt/couchbase/lib/cbas/repo/hyracks-net.jar:/opt/couchbase/lib/cbas/repo/hyracks-storage-am-bloomfilter.jar:/opt/couchbase/lib/cbas/repo/hyracks-storage-am-btree.jar:/opt/couchbase/lib/cbas/repo/hyracks-storage-am-common.jar:/opt/couchbase/lib/cbas/repo/hyracks-storage-am-lsm-btree.jar:/opt/couchbase/lib/cbas/repo/hyracks-storage-am-lsm-common.jar:/opt/couchbase/lib/cbas/repo/hyracks-storage-am-lsm-invertedindex.jar:/opt/couchbase/lib/cbas/repo/hyracks-storage-am-lsm-rtree.jar:/opt/couchbase/lib/cbas/repo/hyracks-storage-am-rtree.jar:/opt/couchbase/lib/cbas/repo/hyracks-storage-common.jar:/opt/couchbase/lib/cbas/repo/hyracks-util.jar:/opt/couchbase/lib/cbas/repo/ini4j-0.5.4.jar:/opt/couchbase/lib/cbas/repo/jackson-annotations-2.8.4.jar:/opt/couchbase/lib/cbas/repo/jackson-core-2.8.4.jar:/opt/couchbase/lib/cbas/repo/jackson-databind-2.8.4.jar:/opt/couchbase/lib/cbas/repo/log4j-api-2.10.0.jar:/opt/couchbase/lib/cbas/repo/log4j-core-2.10.0.jar:/opt/couchbase/lib/cbas/repo/log4j-jcl-2.11.1.jar:/opt/couchbase/lib/cbas/repo/log4j-jul-2.10.0.jar:/opt/couchbase/lib/cbas/repo/log4j-slf4j-impl-2.10.0.jar:/opt/couchbase/lib/cbas/repo/log4j-web-2.10.0.jar:/opt/couchbase/lib/cbas/repo/netty-all-4.1.25.Final.jar:/opt/couchbase/lib/cbas/repo/netty-tcnative-boringssl-static-2.0.8.Final.jar:/opt/couchbase/lib/cbas/repo/opentracing-api-0.31.0.jar:/opt/couchbase/lib/cbas/repo/rxjava-1.3.7.jar:/opt/couchbase/lib/cbas/repo/slf4j-api-1.8.0-alpha2.jar com.couchbase.analytics.control.AnalyticsDriver --jsonconfig /data/@analytics/v_iodevice_0/config.json --nc -node-id 9f255fa768025c9889e50d617ce11a24 --cc]

2018-10-31T01:23:48.030-07:00 INFO CBAS.cbas analytics driver is running; pid: 14771

2018-10-31T01:23:50.054-07:00 FATA CBAS.cbas error cbauth.SetRequestAuth(): Unable to find given hostport in cbauth database: `172.23.106.188:9110'

2018-10-31 01:23:50,612 main ERROR DefaultRolloverStrategy contains an invalid element or attribute "ascending"

2018-10-31 01:23:50,652 main ERROR DefaultRolloverStrategy contains an invalid element or attribute "ascending"

2018-10-31 01:23:50,658 main ERROR DefaultRolloverStrategy contains an invalid element or attribute "ascending"

2018-10-31 01:23:50,666 main ERROR DefaultRolloverStrategy contains an invalid element or attribute "ascending"

2018-10-31 01:23:50,672 main ERROR DefaultRolloverStrategy contains an invalid element or attribute "ascending"

2018-10-31T01:24:03.273-07:00, ns_cluster:1:info:message(ns_1@172.23.106.188) - Node 'ns_1@172.23.106.188' is leaving cluster.

2018-10-31T01:24:03.310-07:00, ns_orchestrator:0:info:message(ns_1@172.23.108.103) - Rebalance completed successfully.

2018-10-31T01:24:04.713-07:00, ns_log:0:info:message(ns_1@172.23.106.188) - Service 'cbas' exited with status 137. Restarting. Messages:

2018-10-31T01:23:52.007-07:00 INFO CBAS.cbas initializing audit service at: http://127.0.0.1:8091

2018-10-31T01:23:53.068-07:00 [INFO] Using plain authentication for user <ud>@cbas</ud> 

2018-10-31T01:23:53.071-07:00 [INFO] Using plain authentication for user <ud>@cbas</ud> 

2018-10-31T01:23:53.072-07:00 [INFO] Using plain authentication for user <ud>@cbas</ud> 

2018-10-31T01:23:53.076-07:00 [INFO] Using plain authentication for user <ud>@cbas</ud> 

2018-10-31T01:23:53.077-07:00 [INFO] Using plain authentication for user <ud>@cbas</ud> 

2018-10-31T01:23:53.077-07:00 INFO CBAS.cbas audit: created new audit service

2018-10-31T01:23:53.085-07:00 INFO CBAS.cbas setting java.home to bundled jre (/opt/couchbase/lib/cbas/runtime)

2018-10-31T01:23:53.270-07:00 WARN CBAS.cbas TLS config has been refreshed by ns server

2018-10-31T01:23:53.284-07:00 INFO CBAS.cbas ignoring already ejected state...

2018-10-31T01:23:53.285-07:00 INFO CBAS.cbas ignoring topology (rebalance) refresh c1f8be96e58de578470d78d21665d690/8 as my node is not included within it (pre-eject, recover, or add)

2018-10-31T01:24:04.721-07:00, ns_log:0:info:message(ns_1@172.23.106.188) - Service 'goxdcr' exited with status 137. Restarting. Messages:

2018-10-31T01:23:17.656-07:00 INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":1038128,"TotalAlloc":45557624,"Sys":9509112,"Lookups":30,"Mallocs":675975,"Frees":665620,"HeapAlloc":1038128,"HeapSys":5472256,"HeapIdle":3301376,"HeapInuse":2170880,"HeapReleased":3112960,"HeapObjects":10355,"StackInuse":819200,"StackSys":819200,"MSpanInuse":43928,"MSpanSys":98304,"MCacheInuse":4800,"MCacheSys":16384,"BuckHashSys":1451421,"GCSys":436224,"OtherSys":1215323,"NextGC":4194304,"LastGC":1540974123164763353,"PauseTotalNs":374632062,"PauseNs":[233035,180895,180998,255562,367388,129312,714961,113937,225025,153587,144311,138844,155957,216101,789798,542488,321726,223448,205423,277816,166524,706212,585520,165755,485018,3695138,336389,321050,260638,196222,135779,131524,585897,28085021,97112,664309,2579049,2368590,2302450,1741864,2775385,941648,3875988,1565680,6060956,1658161,10655418,1128777,8175024,229091,143898,3872591,5225898,124990,22030240,3935005,1409969,1175135,4862823,7003307,2811710,1152268,3736023,1488890,651978,3977067,13896819,704791,511119,2868065,14413296,1972958,154431,5744844,161189,4465541,14641515,369737,89731,6257310,9546545,292258,155511,4670084,389697,806742,15676212,696338,164658,1286667,965443,1055896,511509,993113,1554642,1178844,1604207,785118,334038,1316659,310196,12084579,19958835,1264386,222748,158292,2598643,1233643,18152703,4817046,19690464,2303935,148061,11976615,17627796,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"PauseEnd":[1540960412640216330,1540960533112561957,1540960653140047142,1540960773211968910,1540960893261764908,1540961013311542291,1540961133362436564,1540961253420109984,1540961373472314064,1540961493527314996,1540961613579498209,1540961733634000452,1540961853639973278,1540961973736770797,1540962093789315123,1540962213840357243,1540962333896456487,1540962453955541731,1540962574015993928,1540962694076814690,1540962814132183223,1540962934190280229,1540963054250286846,1540963174309259328,1540963294406309906,1540963414511812807,1540963534613421545,1540963654690587547,1540963774773343805,1540963894846231491,1540964014916542655,1540964134967670563,1540964255037072546,1540964375508875142,1540964495646164041,1540964616153050797,1540964736649543149,1540964857203326882,1540964977477238566,1540965097608770835,1540965218248032692,1540965338638719077,1540965458645996825,1540965579142582838,1540965699431106366,1540965819643789860,1540965939672725419,1540966060354390925,1540966180637265828,1540966300973242178,1540966421294207055,1540966541625592107,1540966661648553007,1540966782132895272,1540966902175689123,1540967022474323878,1540967142605808824,1540967262642612800,1540967382652134416,1540967503670438689,1540967623907735647,1540967744088154638,1540967864344218889,1540967984573612988,1540968104651595239,1540968225146732963,1540968345488875531,1540968465656964233,1540968586635773664,1540968706790411722,1540968827059602148,1540968947268884856,1540969067478535548,1540969187495246474,1540969307619825177,1540969427671551754,1540969548175571798,1540969668641759014,1540969788648546954,1540969908656258593,1540970030651839302,1540970151188555300,1540970271657850212,1540970392482046370,1540970512633406072,1540970633641183415,1540970753670839806,1540970874143017406,1540970994642283149,1540971115457891282,1540971235674567530,1540971355846741826,1540971476018830029,1540971596172575029,1540971716323774532,1540971836518924846,1540971956679400411,1540972077496996762,1540972197608352314,1540972317644936169,1540972438310578450,1540972558716468329,1540972679180719028,1540972799646234105,1540972920153174714,1540973040641037678,1540973161117348523,1540973281474918048,1540973401828732789,1540973522179597599,1540973642528353188,1540973762609775500,1540973882644078878,1540974002666384264,1540974123164763353,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"NumGC":115,"NumForcedGC":0,"GCCPUFraction":0.00002764920010902089,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":9349,"Frees":9257},{"Size":16,"Mallocs":227852,"Frees":225963},{"Size":32,"Mallocs":94184,"Frees":89421},{"Size":48,"Mallocs":131112,"Frees":130017},{"Size":64,"Mallocs":37448,"Frees":36982},{"Size":80,"Mallocs":10169,"Frees":9573},{"Size":96,"Mallocs":37490,"Frees":37148},{"Size":112,"Mallocs":9296,"Frees":9141},{"Size":128,"Mallocs":8844,"Frees":8779},{"Size":144,"Mallocs":235,"Frees":185},{"Size":160,"Mallocs":121,"Frees":82},{"Size":176,"Mallocs":8965,"Frees":8890},{"Size":192,"Mallocs":122,"Frees":120},{"Size":208,"Mallocs":9816,"Frees":9737},{"Size":224,"Mallocs":8730,"Frees":8683},{"Size":240,"Mallocs":2,"Frees":0},{"Size":256,"Mallocs":8791,"Frees":8726},{"Size":288,"Mallocs":213,"Frees":191},{"Size":320,"Mallocs":61,"Frees":57},{"Size":352,"Mallocs":26311,"Frees":26166},{"Size":384,"Mallocs":4,"Frees":1},{"Size":416,"Mallocs":218,"Frees":8},{"Size":448,"Mallocs":9,"Frees":4},{"Size":480,"Mallocs":47,"Frees":44},{"Size":512,"Mallocs":14,"Frees":10},{"Size":576,"Mallocs":138,"Frees":132},{"Size":640,"Mallocs":19,"Frees":12},{"Size":704,"Mallocs":534,"Frees":523},{"Size":768,"Mallocs":46,"Frees":45},{"Size":896,"Mallocs":15,"Frees":8},{"Size":1024,"Mallocs":14,"Frees":4},{"Size":1152,"Mallocs":179,"Frees":173},{"Size":1280,"Mallocs":12,"Frees":7},{"Size":1408,"Mallocs":3,"Frees":2},{"Size":1536,"Mallocs":46,"Frees":46},{"Size":1792,"Mallocs":8,"Frees":3},{"Size":2048,"Mallocs":424,"Frees":402},{"Size":2304,"Mallo