Details
-
Bug
-
Resolution: Won't Fix
-
Major
-
4.5.0
-
4.5.0-2083
-
Untriaged
-
Windows 64-bit
-
Unknown
Description
viber system tests on windows
steps:
Setup 3 nodes cluster as cluster SRC.(172.23.105.143, 172.23.105.94, 172.23.106.29)
Setup 3 nodes cluster as cluster DEST.(172.23.105.248, 172.23.107.96, 172.23.106.83)
At each cluster, create 4 buckets
AbRegNums (0.5GB), MsgsCalls (0.5GB), RevAB (4.5GB) and UserInfo (0.5GB)
1) viber workload load data to cluster SRC in 24 hours.
2) After running load in more than day I started setup replication from cluster SRC to cluster DEST.
3) rebalance in 172.23.107.48 at cluster SRC
rebalance in 172.23.107.227 at cluster DEST
result on SRC cluster:
Resetting rebalance status since it's not really running ns_janitor 000 ns_1@172.23.105.94 3:33:48 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:33:34.317-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale. Was never updated yet., num_of_retry=3
|
MetadataService 2016-04-12T03:33:34.317-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale. Was never updated yet., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:33:34.317-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:33:34 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:34 AM Tue Apr 12, 2016
|
Bucket "AbRegNums" loaded on node 'ns_1@172.23.105.143' in 0 seconds. ns_memcached 000 ns_1@172.23.105.143 3:33:34 AM Tue Apr 12, 2016
|
Bucket "MsgsCalls" loaded on node 'ns_1@172.23.105.143' in 0 seconds. ns_memcached 000 ns_1@172.23.105.143 3:33:33 AM Tue Apr 12, 2016
|
Bucket "RevAB" loaded on node 'ns_1@172.23.105.143' in 0 seconds. ns_memcached 000 ns_1@172.23.105.143 3:33:33 AM Tue Apr 12, 2016
|
Bucket "UserInfo" loaded on node 'ns_1@172.23.105.143' in 0 seconds. ns_memcached 000 ns_1@172.23.105.143 3:33:32 AM Tue Apr 12, 2016
|
Couchbase Server has started on web port 8091 on node 'ns_1@172.23.105.143'. Version: "4.5.0-2083-enterprise". menelaus_sup 001 ns_1@172.23.105.143 3:33:28 AM Tue Apr 12, 2016
|
Node 'ns_1@172.23.106.29' saw that node 'ns_1@172.23.105.143' came up. Tags: [] ns_node_disco 004 ns_1@172.23.106.29 3:33:27 AM Tue Apr 12, 2016
|
Node 'ns_1@172.23.105.143' saw that node 'ns_1@172.23.106.29' came up. Tags: [] ns_node_disco 004 ns_1@172.23.105.143 3:33:27 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:33:24.085-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:33:24.086-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:33:24.086-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:33:24 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:24 AM Tue Apr 12, 2016
|
Node 'ns_1@172.23.107.48' saw that node 'ns_1@172.23.105.143' came up. Tags: [] ns_node_disco 004 ns_1@172.23.107.48 3:33:24 AM Tue Apr 12, 2016
|
Node 'ns_1@172.23.105.94' saw that node 'ns_1@172.23.105.143' came up. Tags: [] ns_node_disco 004 ns_1@172.23.105.94 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:33:17.842-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:33:17.842-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:33:17.843-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:33:17 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:33:11.468-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:33:11.468-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:33:11.468-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:33:11 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:33:05.183-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:33:05.184-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:33:05.184-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:33:05 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:32:58.873-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:32:58.873-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:32:58.873-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:32:58 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:32:52.567-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:32:52.567-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:32:52.567-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:32:52 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'ns_server' exited with status 1. Restarting. Messages: [os_mon] win32 supervisor port (win32sysinfo): Error writing to pipe
|
{"Kernel pid terminated",application_controller,"{application_terminated,os_mon,shutdown}"}
|
|
|
Crash dump was written to: erl_crash.dump
|
Kernel pid terminated (application_controller) ({application_terminated,os_mon,shutdown}) ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:32:46.323-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:32:46.323-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:32:46.323-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:32:46 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Node 'ns_1@172.23.105.143' synchronized otp cookie iijwwhllwkycpcbp from cluster ns_cookie_manager 002 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:32:39.981-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:32:39.982-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:32:39.982-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:32:40 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:32:33.751-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:32:33.751-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:32:33.751-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:32:33 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:32:27.493-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:32:27.493-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:32:27.493-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:32:27 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:32:21.257-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:32:21.257-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:32:21.257-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:32:21 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:32:15.062-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:32:15.062-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:32:15.062-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:32:15 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:32:08.795-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:32:08.795-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:32:08.795-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:32:08 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:32:02.487-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:32:02.487-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:32:02.487-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:32:02 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:31:56.253-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:31:56.253-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:31:56.253-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:31:56 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:31:49.983-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:31:49.983-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:31:49.983-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:31:49 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:31:43.756-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:31:43.756-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:31:43.756-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:31:43 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:31:37.487-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:31:37.487-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:31:37.487-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:31:37 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:31:31.150-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:31:31.150-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:31:31.150-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:31:31 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:22 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:31:24.848-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:31:24.848-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:31:24.848-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:31:24 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:21 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:31:18.561-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:31:18.561-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:31:18.561-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:31:18 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:21 AM Tue Apr 12, 2016
|
Service 'goxdcr' exited with status 1. Restarting. Messages: MetadataService 2016-04-12T03:31:12.032-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=3
|
MetadataService 2016-04-12T03:31:12.032-07:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connectex: No connection could be made because the target machine actively refused it., num_of_retry=4
|
RemoteClusterService 2016-04-12T03:31:12.032-07:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
|
Error starting remote cluster service. err=metakv failed for max number of retries = 5
|
[goport] 2016/04/12 03:31:12 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 1 ns_log 000 ns_1@172.23.105.143 3:33:21 AM Tue Apr 12, 2016
|
Node 'ns_1@172.23.105.94' saw that node 'ns_1@172.23.105.143' went down. Details: [{nodedown_reason,
|
connection_closed}] ns_node_disco 005 ns_1@172.23.105.94 3:32:48 AM Tue Apr 12, 2016
|
Node 'ns_1@172.23.107.48' saw that node 'ns_1@172.23.105.143' went down. Details: [{nodedown_reason,
|
connection_closed}] ns_node_disco 005 ns_1@172.23.107.48 3:32:48 AM Tue Apr 12, 2016
|
Node 'ns_1@172.23.106.29' saw that node 'ns_1@172.23.105.143' went down. Details: [{nodedown_reason,
|
connection_closed}] ns_node_disco 005 ns_1@172.23.106.29 3:32:48 AM Tue Apr 12, 2016
|
Shutting down bucket "AbRegNums" on 'ns_1@172.23.105.143' for server shutdown ns_memcached 000 ns_1@172.23.105.143 3:31:00 AM Tue Apr 12, 2016
|
Shutting down bucket "MsgsCalls" on 'ns_1@172.23.105.143' for server shutdown ns_memcached 000 ns_1@172.23.105.143 3:30:56 AM Tue Apr 12, 2016
|
Shutting down bucket "RevAB" on 'ns_1@172.23.105.143' for server shutdown ns_memcached 000 ns_1@172.23.105.143 3:30:52 AM Tue Apr 12, 2016
|
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master 000 ns_1@172.23.105.94 3:30:49 AM Tue Apr 12, 2016
|
Shutting down bucket "UserInfo" on 'ns_1@172.23.105.143' for server shutdown ns_memcached 000 ns_1@172.23.105.143 3:30:49 AM Tue Apr 12, 2016
|
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master 000 ns_1@172.23.105.94 3:29:53 AM Tue Apr 12, 2016
|
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master 000 ns_1@172.23.105.94 3:22:45 AM Tue Apr 12, 2016
|
Bucket "RevAB" rebalance does not seem to be swap rebalance ns_vbucket_mover 000 ns_1@172.23.105.143 3:07:10 AM Tue Apr 12, 2016
|
Bucket "RevAB" loaded on node 'ns_1@172.23.107.48' in 0 seconds. ns_memcached 000 ns_1@172.23.107.48 3:07:06 AM Tue Apr 12, 2016
|
Started rebalancing bucket RevAB ns_rebalancer 000 ns_1@172.23.105.143 3:07:05 AM Tue Apr 12, 2016
|
Bucket "UserInfo" rebalance does not seem to be swap rebalance ns_vbucket_mover 000 ns_1@172.23.105.143 2:58:25 AM Tue Apr 12, 2016
|
Bucket "UserInfo" loaded on node 'ns_1@172.23.107.48' in 0 seconds. ns_memcached 000 ns_1@172.23.107.48 2:58:19 AM Tue Apr 12, 2016
|
Started rebalancing bucket UserInfo ns_rebalancer 000 ns_1@172.23.105.143 2:58:19 AM Tue Apr 12, 2016
|
Starting rebalance, KeepNodes = ['ns_1@172.23.105.143','ns_1@172.23.105.94',
|
'ns_1@172.23.106.29','ns_1@172.23.107.48'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes
|
ns_orchestrator 004 ns_1@172.23.105.143 2:58:18 AM Tue Apr 12, 2016
|
https://s3.amazonaws.com/bugdb/jira/MB-19166/2419cdd2/172.23.105.143-20160412-0417-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-19166/2419cdd2/172.23.105.94-20160412-0430-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-19166/2419cdd2/172.23.106.29-20160412-0445-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-19166/2419cdd2/172.23.107.48-20160412-0458-diag.zip
