[2020-04-07 20:06:54,857] - [rest_client] [140562577667840] - INFO - /diag/eval status on 172.23.98.17:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2020-04-07 20:06:54,858] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.17 with username:root [2020-04-07 20:06:54,956] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.17 [2020-04-07 20:06:55,227] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:06:55,626] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:06:55,627] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.17: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:06:55,723] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:06:55,837] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:06:55,838] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.17 with username:root [2020-04-07 20:06:55,935] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.17 [2020-04-07 20:06:56,201] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:06:56,600] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:06:56,600] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.17: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:06:56,696] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:06:56,810] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:06:56,811] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.164 with username:root [2020-04-07 20:06:56,911] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.164 [2020-04-07 20:06:57,176] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:06:57,739] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:06:57,740] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.164: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:06:57,773] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:06:57,887] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:06:57,888] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.164 with username:root [2020-04-07 20:06:57,985] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.164 [2020-04-07 20:06:58,225] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:06:58,608] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:06:58,608] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.164: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:06:58,705] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:06:58,818] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:06:58,820] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.165 with username:root [2020-04-07 20:06:58,920] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.165 [2020-04-07 20:06:59,163] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:06:59,563] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:06:59,563] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:06:59,644] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:06:59,757] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:06:59,759] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.165 with username:root [2020-04-07 20:06:59,856] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.165 [2020-04-07 20:07:00,423] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:00,825] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:00,826] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:07:00,906] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:01,020] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:07:01,022] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.16 with username:root [2020-04-07 20:07:01,121] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.16 [2020-04-07 20:07:01,363] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:01,738] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:01,738] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.16: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:07:01,819] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:01,932] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:07:01,934] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.16 with username:root [2020-04-07 20:07:02,031] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.16 [2020-04-07 20:07:02,274] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:02,676] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:02,676] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.16: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:07:02,757] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:02,871] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:07:02,873] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.15 with username:root [2020-04-07 20:07:02,974] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.15 [2020-04-07 20:07:03,246] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:03,676] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:03,677] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.15: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:07:03,757] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:03,871] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:07:03,872] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.15 with username:root [2020-04-07 20:07:03,969] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.15 [2020-04-07 20:07:04,216] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:04,618] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:04,618] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.15: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:07:04,698] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:04,812] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:07:04,813] - [remote_util] [140562577667840] - INFO - connecting to 172.23.97.93 with username:root [2020-04-07 20:07:04,946] - [remote_util] [140562577667840] - INFO - Connected to 172.23.97.93 [2020-04-07 20:07:05,184] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:05,583] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:05,583] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.97.93: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:07:05,663] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:05,777] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:07:05,778] - [remote_util] [140562577667840] - INFO - connecting to 172.23.97.93 with username:root [2020-04-07 20:07:05,874] - [remote_util] [140562577667840] - INFO - Connected to 172.23.97.93 [2020-04-07 20:07:06,118] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:06,541] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:06,542] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.97.93: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:07:06,622] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:06,736] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:07:06,736] - [xdcrnewbasetests] [140562577667840] - INFO - {'172.23.97.93': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.98.17': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.98.16': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.98.15': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.98.164': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.98.165': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}} [2020-04-07 20:07:06,745] - [rest_client] [140562577667840] - INFO - /diag/eval status on 172.23.98.17:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2020-04-07 20:07:06,746] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.17 with username:root [2020-04-07 20:07:06,846] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.17 [2020-04-07 20:07:07,091] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:07,443] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:07,444] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.17: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:07:07,540] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:07,654] - [xdcrnewbasetests] [140562577667840] - INFO - 1 [2020-04-07 20:07:07,656] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.164 with username:root [2020-04-07 20:07:07,756] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.164 [2020-04-07 20:07:07,999] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:08,389] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:08,390] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.164: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:07:08,486] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:08,600] - [xdcrnewbasetests] [140562577667840] - INFO - 4 [2020-04-07 20:07:08,602] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.165 with username:root [2020-04-07 20:07:08,702] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.165 [2020-04-07 20:07:08,969] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:09,336] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:09,337] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:07:09,417] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:09,531] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:07:09,532] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.16 with username:root [2020-04-07 20:07:09,631] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.16 [2020-04-07 20:07:09,873] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:10,263] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:10,264] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.16: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:07:10,344] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:10,458] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:07:10,459] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.15 with username:root [2020-04-07 20:07:10,559] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.15 [2020-04-07 20:07:10,834] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:11,238] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:11,238] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.15: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:07:11,319] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:11,433] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:07:11,434] - [remote_util] [140562577667840] - INFO - connecting to 172.23.97.93 with username:root [2020-04-07 20:07:11,535] - [remote_util] [140562577667840] - INFO - Connected to 172.23.97.93 [2020-04-07 20:07:11,782] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:12,183] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:12,184] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.97.93: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2020-04-07 20:07:12,264] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:12,378] - [xdcrnewbasetests] [140562577667840] - INFO - 0 [2020-04-07 20:07:12,378] - [xdcrnewbasetests] [140562577667840] - INFO - {'172.23.97.93': 0, '172.23.98.17': 1, '172.23.98.16': 0, '172.23.98.15': 0, '172.23.98.164': 4, '172.23.98.165': 0} [2020-04-07 20:07:12,378] - [xdcrnewbasetests] [140562577667840] - INFO - ==== XDCRNewbasetests setup is started for test #10 load_with_async_ops_with_warmup_master ==== [2020-04-07 20:07:12,379] - [xdcrnewbasetests] [140562577667840] - INFO - removing xdcr/nodes settings [2020-04-07 20:07:12,404] - [xdcrnewbasetests] [140562577667840] - INFO - cleanup [ip:172.23.98.17 port:8091 ssh_username:root, ip:172.23.98.164 port:8091 ssh_username:root] [2020-04-07 20:07:12,433] - [cluster_helper] [140562577667840] - INFO - waiting for ns_server @ 172.23.98.17:8091 [2020-04-07 20:07:12,438] - [cluster_helper] [140562577667840] - INFO - ns_server @ 172.23.98.17:8091 is running [2020-04-07 20:07:12,466] - [cluster_helper] [140562577667840] - INFO - waiting for ns_server @ 172.23.98.164:8091 [2020-04-07 20:07:12,470] - [cluster_helper] [140562577667840] - INFO - ns_server @ 172.23.98.164:8091 is running [2020-04-07 20:07:12,470] - [xdcrnewbasetests] [140562577667840] - INFO - removing xdcr/nodes settings [2020-04-07 20:07:12,496] - [xdcrnewbasetests] [140562577667840] - INFO - cleanup [ip:172.23.98.165 port:8091 ssh_username:root, ip:172.23.98.16 port:8091 ssh_username:root] [2020-04-07 20:07:12,525] - [cluster_helper] [140562577667840] - INFO - waiting for ns_server @ 172.23.98.165:8091 [2020-04-07 20:07:12,530] - [cluster_helper] [140562577667840] - INFO - ns_server @ 172.23.98.165:8091 is running [2020-04-07 20:07:12,558] - [cluster_helper] [140562577667840] - INFO - waiting for ns_server @ 172.23.98.16:8091 [2020-04-07 20:07:12,562] - [cluster_helper] [140562577667840] - INFO - ns_server @ 172.23.98.16:8091 is running [2020-04-07 20:07:12,562] - [xdcrnewbasetests] [140562577667840] - INFO - Initializing all clusters... [2020-04-07 20:07:12,568] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.17 with username:root [2020-04-07 20:07:12,667] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.17 [2020-04-07 20:07:12,936] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:13,328] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:13,339] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.17: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-04-07 20:07:13,403] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:13,403] - [xdcrnewbasetests] [140562577667840] - INFO - Enabled diag/eval for non-local hosts from 172.23.98.17 [2020-04-07 20:07:13,520] - [rest_client] [140562577667840] - INFO - /diag/eval status on 172.23.98.17:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-04-07 20:07:14,391] - [task] [140561845823232] - INFO - server: ip:172.23.98.17 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.98.17', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.98.17', 'uptime': u'1731', 'mcdMemoryReserved': 3310, 'storageTotalRam': 4138, 'hostname': u'172.23.98.17:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'7.0.0-1757-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 458752, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3558326272, 'memoryTotal': 4339109888, 'memoryQuota': 2217, 'mcdMemoryAllocated': 3310, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-04-07 20:07:14,396] - [rest_client] [140561845823232] - INFO - pools/default params : memoryQuota=2217 [2020-04-07 20:07:14,401] - [rest_client] [140561845823232] - INFO - settings/web params on 172.23.98.17:8091:username=Administrator&password=password&port=8091 [2020-04-07 20:07:14,436] - [remote_util] [140561845823232] - INFO - connecting to 172.23.98.17 with username:root [2020-04-07 20:07:14,535] - [remote_util] [140561845823232] - INFO - Connected to 172.23.98.17 [2020-04-07 20:07:14,780] - [remote_util] [140561845823232] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:15,140] - [remote_util] [140561845823232] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:15,152] - [remote_util] [140561845823232] - INFO - running command.raw on 172.23.98.17: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-04-07 20:07:15,216] - [remote_util] [140561845823232] - INFO - command executed successfully [2020-04-07 20:07:15,220] - [rest_client] [140561845823232] - INFO - /diag/eval status on 172.23.98.17:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-04-07 20:07:15,220] - [rest_client] [140561845823232] - INFO - settings/indexes params : storageMode=forestdb [2020-04-07 20:07:15,224] - [rest_client] [140561845823232] - ERROR - POST http://172.23.98.17:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2020-04-07 20:07:15,356] - [task] [140561845823232] - INFO - server: ip:172.23.98.164 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.98.164', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.98.164', 'uptime': u'2211', 'mcdMemoryReserved': 3310, 'storageTotalRam': 4138, 'hostname': u'172.23.98.164:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'7.0.0-1757-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 458752, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3806687232, 'memoryTotal': 4339109888, 'memoryQuota': 1817, 'mcdMemoryAllocated': 3310, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-04-07 20:07:15,359] - [rest_client] [140561845823232] - ERROR - GET http://172.23.98.164:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-04-07 20:07:15,359] - [rest_client] [140561845823232] - INFO - pools/default params : memoryQuota=2217 [2020-04-07 20:07:15,364] - [rest_client] [140561845823232] - INFO - settings/web params on 172.23.98.164:8091:username=Administrator&password=password&port=8091 [2020-04-07 20:07:15,398] - [remote_util] [140561845823232] - INFO - connecting to 172.23.98.164 with username:root [2020-04-07 20:07:15,496] - [remote_util] [140561845823232] - INFO - Connected to 172.23.98.164 [2020-04-07 20:07:15,764] - [remote_util] [140561845823232] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:16,179] - [remote_util] [140561845823232] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:16,189] - [remote_util] [140561845823232] - INFO - running command.raw on 172.23.98.164: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-04-07 20:07:16,254] - [remote_util] [140561845823232] - INFO - command executed successfully [2020-04-07 20:07:16,257] - [rest_client] [140561845823232] - INFO - /diag/eval status on 172.23.98.164:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-04-07 20:07:16,257] - [rest_client] [140561845823232] - INFO - settings/indexes params : storageMode=forestdb [2020-04-07 20:07:16,260] - [rest_client] [140561845823232] - ERROR - POST http://172.23.98.164:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2020-04-07 20:07:17,384] - [task] [140561845823232] - INFO - adding node 172.23.98.164:8091 to cluster [2020-04-07 20:07:17,384] - [rest_client] [140561845823232] - INFO - adding remote node @172.23.98.164:8091 to this cluster @172.23.98.17:8091 [2020-04-07 20:07:18,888] - [rest_client] [140561845823232] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.98.17,ns_1@172.23.98.164'} [2020-04-07 20:07:18,894] - [rest_client] [140561845823232] - INFO - rebalance operation started [2020-04-07 20:07:18,902] - [rest_client] [140561845823232] - INFO - rebalance percentage : 0.00 % [2020-04-07 20:07:18,902] - [task] [140561845823232] - INFO - Rebalance - status: running, progress: 0.00% [2020-04-07 20:07:28,928] - [task] [140561845823232] - INFO - Rebalance - status: none, progress: 100.00% [2020-04-07 20:07:28,935] - [task] [140561845823232] - INFO - rebalancing was completed with progress: 100% in 10.0409331322 sec [2020-04-07 20:07:28,942] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.165 with username:root [2020-04-07 20:07:29,043] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.165 [2020-04-07 20:07:29,312] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:29,705] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:29,718] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-04-07 20:07:29,782] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:07:29,782] - [xdcrnewbasetests] [140562577667840] - INFO - Enabled diag/eval for non-local hosts from 172.23.98.165 [2020-04-07 20:07:29,899] - [rest_client] [140562577667840] - INFO - /diag/eval status on 172.23.98.165:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-04-07 20:07:30,409] - [task] [140561871001344] - INFO - server: ip:172.23.98.165 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.98.165', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.98.165', 'uptime': u'939', 'mcdMemoryReserved': 3310, 'storageTotalRam': 4138, 'hostname': u'172.23.98.165:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'7.0.0-1757-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 458752, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3643297792, 'memoryTotal': 4339109888, 'memoryQuota': 2217, 'mcdMemoryAllocated': 3310, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-04-07 20:07:30,414] - [rest_client] [140561871001344] - INFO - pools/default params : memoryQuota=2217 [2020-04-07 20:07:30,422] - [rest_client] [140561871001344] - INFO - settings/web params on 172.23.98.165:8091:username=Administrator&password=password&port=8091 [2020-04-07 20:07:30,458] - [remote_util] [140561871001344] - INFO - connecting to 172.23.98.165 with username:root [2020-04-07 20:07:30,554] - [remote_util] [140561871001344] - INFO - Connected to 172.23.98.165 [2020-04-07 20:07:30,797] - [remote_util] [140561871001344] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:31,180] - [remote_util] [140561871001344] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:31,191] - [remote_util] [140561871001344] - INFO - running command.raw on 172.23.98.165: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-04-07 20:07:31,256] - [remote_util] [140561871001344] - INFO - command executed successfully [2020-04-07 20:07:31,259] - [rest_client] [140561871001344] - INFO - /diag/eval status on 172.23.98.165:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-04-07 20:07:31,259] - [rest_client] [140561871001344] - INFO - settings/indexes params : storageMode=forestdb [2020-04-07 20:07:31,265] - [rest_client] [140561871001344] - ERROR - POST http://172.23.98.165:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2020-04-07 20:07:31,398] - [task] [140561871001344] - INFO - server: ip:172.23.98.16 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.98.16', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.98.16', 'uptime': u'2213', 'mcdMemoryReserved': 3310, 'storageTotalRam': 4138, 'hostname': u'172.23.98.16:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'7.0.0-1757-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 458752, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3785203712, 'memoryTotal': 4339109888, 'memoryQuota': 1817, 'mcdMemoryAllocated': 3310, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-04-07 20:07:31,402] - [rest_client] [140561871001344] - ERROR - GET http://172.23.98.16:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-04-07 20:07:31,402] - [rest_client] [140561871001344] - INFO - pools/default params : memoryQuota=2217 [2020-04-07 20:07:31,407] - [rest_client] [140561871001344] - INFO - settings/web params on 172.23.98.16:8091:username=Administrator&password=password&port=8091 [2020-04-07 20:07:31,443] - [remote_util] [140561871001344] - INFO - connecting to 172.23.98.16 with username:root [2020-04-07 20:07:31,541] - [remote_util] [140561871001344] - INFO - Connected to 172.23.98.16 [2020-04-07 20:07:31,806] - [remote_util] [140561871001344] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:07:32,198] - [remote_util] [140561871001344] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:07:32,208] - [remote_util] [140561871001344] - INFO - running command.raw on 172.23.98.16: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-04-07 20:07:32,273] - [remote_util] [140561871001344] - INFO - command executed successfully [2020-04-07 20:07:32,277] - [rest_client] [140561871001344] - INFO - /diag/eval status on 172.23.98.16:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-04-07 20:07:32,277] - [rest_client] [140561871001344] - INFO - settings/indexes params : storageMode=forestdb [2020-04-07 20:07:32,280] - [rest_client] [140561871001344] - ERROR - POST http://172.23.98.16:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2020-04-07 20:07:33,404] - [task] [140561871001344] - INFO - adding node 172.23.98.16:8091 to cluster [2020-04-07 20:07:33,404] - [rest_client] [140561871001344] - INFO - adding remote node @172.23.98.16:8091 to this cluster @172.23.98.165:8091 [2020-04-07 20:07:35,062] - [rest_client] [140561871001344] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.98.16,ns_1@172.23.98.165'} [2020-04-07 20:07:35,069] - [rest_client] [140561871001344] - INFO - rebalance operation started [2020-04-07 20:07:35,075] - [rest_client] [140561871001344] - INFO - rebalance percentage : 0.00 % [2020-04-07 20:07:35,075] - [task] [140561871001344] - INFO - Rebalance - status: running, progress: 0.00% [2020-04-07 20:07:45,095] - [task] [140561871001344] - INFO - Rebalance - status: none, progress: 100.00% [2020-04-07 20:07:45,102] - [task] [140561871001344] - INFO - rebalancing was completed with progress: 100% in 10.032430172 sec [2020-04-07 20:07:45,122] - [rest_client] [140562577667840] - ERROR - DELETE http://172.23.98.17:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2020-04-07 20:07:45,123] - [internal_user] [140562577667840] - INFO - Exception while deleting user. Exception is -"User was not found." [2020-04-07 20:07:45,201] - [rest_client] [140562577667840] - ERROR - DELETE http://172.23.98.165:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2020-04-07 20:07:45,202] - [internal_user] [140562577667840] - INFO - Exception while deleting user. Exception is -"User was not found." [2020-04-07 20:07:45,977] - [rest_client] [140561845823232] - INFO - http://172.23.98.17:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=858&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 [2020-04-07 20:07:45,984] - [rest_client] [140561845823232] - INFO - 0.01 seconds to create bucket default [2020-04-07 20:07:45,984] - [bucket_helper] [140561845823232] - INFO - waiting for memcached bucket : default in 172.23.98.17 to accept set ops [2020-04-07 20:07:46,583] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:07:46,656] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:07:46,738] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:07:46,805] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:07:46,853] - [task] [140561845823232] - WARNING - vbucket map not ready after try 0 [2020-04-07 20:07:46,853] - [bucket_helper] [140561845823232] - INFO - waiting for memcached bucket : default in 172.23.98.17 to accept set ops [2020-04-07 20:07:46,931] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:07:46,998] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:07:47,083] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:07:47,154] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:07:47,206] - [task] [140561845823232] - WARNING - vbucket map not ready after try 1 [2020-04-07 20:07:47,207] - [bucket_helper] [140561845823232] - INFO - waiting for memcached bucket : default in 172.23.98.17 to accept set ops [2020-04-07 20:07:47,296] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:07:47,356] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:07:47,445] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:07:47,517] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:07:47,568] - [task] [140561845823232] - WARNING - vbucket map not ready after try 2 [2020-04-07 20:07:47,569] - [bucket_helper] [140561845823232] - INFO - waiting for memcached bucket : default in 172.23.98.17 to accept set ops [2020-04-07 20:07:47,679] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:07:47,753] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:07:47,858] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:07:47,919] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:07:47,966] - [task] [140561845823232] - WARNING - vbucket map not ready after try 3 [2020-04-07 20:07:47,966] - [bucket_helper] [140561845823232] - INFO - waiting for memcached bucket : default in 172.23.98.17 to accept set ops [2020-04-07 20:07:48,055] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:07:48,115] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:07:48,212] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:07:48,272] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:07:48,320] - [task] [140561845823232] - WARNING - vbucket map not ready after try 4 [2020-04-07 20:07:48,321] - [bucket_helper] [140561845823232] - INFO - waiting for memcached bucket : default in 172.23.98.17 to accept set ops [2020-04-07 20:07:48,417] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:07:48,476] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:07:48,575] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:07:48,636] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:07:48,694] - [task] [140561845823232] - WARNING - vbucket map not ready after try 5 [2020-04-07 20:07:49,722] - [rest_client] [140561845823232] - INFO - http://172.23.98.17:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=standard_bucket_1&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=858 [2020-04-07 20:07:49,729] - [rest_client] [140561845823232] - INFO - 0.01 seconds to create bucket standard_bucket_1 [2020-04-07 20:07:49,729] - [bucket_helper] [140561845823232] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.98.17 to accept set ops [2020-04-07 20:07:50,352] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:07:50,447] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:07:50,557] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:07:50,638] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:07:50,688] - [task] [140561845823232] - WARNING - vbucket map not ready after try 0 [2020-04-07 20:07:50,688] - [bucket_helper] [140561845823232] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.98.17 to accept set ops [2020-04-07 20:07:50,776] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:07:50,848] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:07:50,938] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:07:51,014] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:07:51,068] - [task] [140561845823232] - WARNING - vbucket map not ready after try 1 [2020-04-07 20:07:51,068] - [bucket_helper] [140561845823232] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.98.17 to accept set ops [2020-04-07 20:07:51,166] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:07:51,230] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:07:51,334] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:07:51,398] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:07:51,447] - [task] [140561845823232] - WARNING - vbucket map not ready after try 2 [2020-04-07 20:07:51,447] - [bucket_helper] [140561845823232] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.98.17 to accept set ops [2020-04-07 20:07:51,544] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:07:51,611] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:07:51,713] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:07:51,782] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:07:51,832] - [task] [140561845823232] - WARNING - vbucket map not ready after try 3 [2020-04-07 20:07:51,833] - [bucket_helper] [140561845823232] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.98.17 to accept set ops [2020-04-07 20:07:51,933] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:07:51,995] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:07:52,097] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:07:52,158] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:07:52,207] - [task] [140561845823232] - WARNING - vbucket map not ready after try 4 [2020-04-07 20:07:52,207] - [bucket_helper] [140561845823232] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.98.17 to accept set ops [2020-04-07 20:07:52,307] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:07:52,369] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:07:52,476] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:07:52,539] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:07:52,607] - [task] [140561845823232] - WARNING - vbucket map not ready after try 5 [2020-04-07 20:07:53,136] - [rest_client] [140561871001344] - INFO - http://172.23.98.165:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=858&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 [2020-04-07 20:07:53,143] - [rest_client] [140561871001344] - INFO - 0.01 seconds to create bucket default [2020-04-07 20:07:53,143] - [bucket_helper] [140561871001344] - INFO - waiting for memcached bucket : default in 172.23.98.165 to accept set ops [2020-04-07 20:07:53,769] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:07:53,858] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:07:53,968] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:07:54,046] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:07:54,096] - [task] [140561871001344] - WARNING - vbucket map not ready after try 0 [2020-04-07 20:07:54,096] - [bucket_helper] [140561871001344] - INFO - waiting for memcached bucket : default in 172.23.98.165 to accept set ops [2020-04-07 20:07:54,179] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:07:54,249] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:07:54,329] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:07:54,397] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:07:54,443] - [task] [140561871001344] - WARNING - vbucket map not ready after try 1 [2020-04-07 20:07:54,443] - [bucket_helper] [140561871001344] - INFO - waiting for memcached bucket : default in 172.23.98.165 to accept set ops [2020-04-07 20:07:54,530] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:07:54,586] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:07:54,664] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:07:54,733] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:07:54,784] - [task] [140561871001344] - WARNING - vbucket map not ready after try 2 [2020-04-07 20:07:54,784] - [bucket_helper] [140561871001344] - INFO - waiting for memcached bucket : default in 172.23.98.165 to accept set ops [2020-04-07 20:07:54,878] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:07:54,938] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:07:55,037] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:07:55,100] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:07:55,148] - [task] [140561871001344] - WARNING - vbucket map not ready after try 3 [2020-04-07 20:07:55,149] - [bucket_helper] [140561871001344] - INFO - waiting for memcached bucket : default in 172.23.98.165 to accept set ops [2020-04-07 20:07:55,242] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:07:55,300] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:07:55,394] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:07:55,451] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:07:55,496] - [task] [140561871001344] - WARNING - vbucket map not ready after try 4 [2020-04-07 20:07:55,497] - [bucket_helper] [140561871001344] - INFO - waiting for memcached bucket : default in 172.23.98.165 to accept set ops [2020-04-07 20:07:55,589] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:07:55,650] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:07:55,745] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:07:55,805] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:07:55,870] - [task] [140561871001344] - WARNING - vbucket map not ready after try 5 [2020-04-07 20:07:56,898] - [rest_client] [140561871001344] - INFO - http://172.23.98.165:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=standard_bucket_1&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=858 [2020-04-07 20:07:56,906] - [rest_client] [140561871001344] - INFO - 0.01 seconds to create bucket standard_bucket_1 [2020-04-07 20:07:56,907] - [bucket_helper] [140561871001344] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.98.165 to accept set ops [2020-04-07 20:07:57,527] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:07:57,612] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:07:57,697] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:07:57,773] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:07:57,821] - [task] [140561871001344] - WARNING - vbucket map not ready after try 0 [2020-04-07 20:07:57,821] - [bucket_helper] [140561871001344] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.98.165 to accept set ops [2020-04-07 20:07:57,903] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:07:57,986] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:07:58,084] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:07:58,167] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:07:58,221] - [task] [140561871001344] - WARNING - vbucket map not ready after try 1 [2020-04-07 20:07:58,221] - [bucket_helper] [140561871001344] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.98.165 to accept set ops [2020-04-07 20:07:58,326] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:07:58,395] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:07:58,502] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:07:58,573] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:07:58,633] - [task] [140561871001344] - WARNING - vbucket map not ready after try 2 [2020-04-07 20:07:58,634] - [bucket_helper] [140561871001344] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.98.165 to accept set ops [2020-04-07 20:07:58,768] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:07:58,840] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:07:58,939] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:07:59,001] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:07:59,052] - [task] [140561871001344] - WARNING - vbucket map not ready after try 3 [2020-04-07 20:07:59,052] - [bucket_helper] [140561871001344] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.98.165 to accept set ops [2020-04-07 20:07:59,151] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:07:59,214] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:07:59,313] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:07:59,374] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:07:59,425] - [task] [140561871001344] - WARNING - vbucket map not ready after try 4 [2020-04-07 20:07:59,425] - [bucket_helper] [140561871001344] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.98.165 to accept set ops [2020-04-07 20:07:59,524] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:07:59,584] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:07:59,684] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:07:59,745] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:07:59,810] - [task] [140561871001344] - WARNING - vbucket map not ready after try 5 [2020-04-07 20:07:59,810] - [xdcrnewbasetests] [140562577667840] - INFO - ==== XDCRNewbasetests setup is finished for test #10 load_with_async_ops_with_warmup_master ==== [2020-04-07 20:07:59,827] - [rest_client] [140562577667840] - INFO - adding remote cluster hostname:172.23.98.165:8091 with username:password Administrator:password name:remote_cluster_C1-C2 to source node: 172.23.98.17:8091 [2020-04-07 20:07:59,835] - [rest_client] [140562577667840] - INFO - Node versions in cluster [u'7.0.0-1757-enterprise', u'7.0.0-1757-enterprise'] [2020-04-07 20:07:59,849] - [rest_client] [140562577667840] - INFO - Node versions in cluster [u'7.0.0-1757-enterprise', u'7.0.0-1757-enterprise'] [2020-04-07 20:08:19,922] - [rest_client] [140562577667840] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {} [2020-04-07 20:08:20,029] - [rest_client] [140562577667840] - INFO - Replication created with id: 32a5a3fe83d9f903bcf55ade47abaa6f/default/default [2020-04-07 20:08:30,045] - [rest_client] [140562577667840] - INFO - starting continuous replication type:xmem from standard_bucket_1 to standard_bucket_1 in the remote cluster remote_cluster_C1-C2 with settings {} [2020-04-07 20:08:30,143] - [rest_client] [140562577667840] - INFO - Replication created with id: 32a5a3fe83d9f903bcf55ade47abaa6f/standard_bucket_1/standard_bucket_1 [2020-04-07 20:08:40,180] - [rest_client] [140562577667840] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.98.17 [2020-04-07 20:08:40,202] - [rest_client] [140562577667840] - INFO - Updated checkpointInterval=60 on bucket'standard_bucket_1' on 172.23.98.17 [2020-04-07 20:08:40,255] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:08:40,318] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:08:40,444] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:08:40,504] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:08:40,758] - [data_helper] [140561854215936] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:08:40,767] - [data_helper] [140562569275136] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:08:40,779] - [data_helper] [140562569275136] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:08:40,789] - [data_helper] [140561854215936] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:08:40,828] - [data_helper] [140561854215936] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:08:40,830] - [data_helper] [140562569275136] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:08:40,846] - [data_helper] [140562569275136] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:08:40,860] - [data_helper] [140561854215936] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:08:45,671] - [data_helper] [140562569275136] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:08:45,679] - [data_helper] [140561854215936] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:08:45,755] - [data_helper] [140562569275136] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:08:45,760] - [data_helper] [140561854215936] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:08:45,820] - [data_helper] [140562569275136] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:08:45,833] - [data_helper] [140561854215936] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:08:45,921] - [data_helper] [140561854215936] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:08:45,923] - [data_helper] [140562569275136] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:08:51,900] - [xdcrnewbasetests] [140562577667840] - INFO - sleep for 10 secs. ... [2020-04-07 20:09:01,913] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.17 with username:root [2020-04-07 20:09:02,013] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.17 [2020-04-07 20:09:02,259] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:09:02,645] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:09:02,748] - [remote_util] [140562577667840] - INFO - Running systemd command on this server [2020-04-07 20:09:02,748] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.17: systemctl stop couchbase-server.service [2020-04-07 20:09:04,717] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:09:09,723] - [remote_util] [140562577667840] - INFO - Checking for process beam.smp on linux [2020-04-07 20:09:09,758] - [remote_util] [140562577667840] - INFO - Starting couchbase server [2020-04-07 20:09:09,859] - [remote_util] [140562577667840] - INFO - Running systemd command on this server [2020-04-07 20:09:09,860] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.17: systemctl start couchbase-server.service [2020-04-07 20:09:09,877] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:09:09,878] - [remote_util] [140562577667840] - INFO - 172.23.98.17:sleep for 5 secs. waiting for couchbase server to come up ... [2020-04-07 20:09:14,883] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.17: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-04-07 20:09:14,965] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:09:14,965] - [remote_util] [140562577667840] - INFO - Couchbase server status: [' Process: 25434 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-04-07 20:09:14,965] - [remote_util] [140562577667840] - INFO - Checking for process beam.smp on linux [2020-04-07 20:09:15,017] - [remote_util] [140562577667840] - INFO - process beam.smp is running on 172.23.98.17: with pid 25512 [2020-04-07 20:09:15,132] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.165 with username:root [2020-04-07 20:09:15,232] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.165 [2020-04-07 20:09:15,500] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:09:15,901] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:09:16,005] - [remote_util] [140562577667840] - INFO - Running systemd command on this server [2020-04-07 20:09:16,005] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: systemctl stop couchbase-server.service [2020-04-07 20:09:17,875] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:09:22,880] - [remote_util] [140562577667840] - INFO - Checking for process beam.smp on linux [2020-04-07 20:09:22,915] - [remote_util] [140562577667840] - INFO - Starting couchbase server [2020-04-07 20:09:23,017] - [remote_util] [140562577667840] - INFO - Running systemd command on this server [2020-04-07 20:09:23,017] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: systemctl start couchbase-server.service [2020-04-07 20:09:23,035] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:09:23,035] - [remote_util] [140562577667840] - INFO - 172.23.98.165:sleep for 5 secs. waiting for couchbase server to come up ... [2020-04-07 20:09:28,040] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-04-07 20:09:28,060] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:09:28,060] - [remote_util] [140562577667840] - INFO - Couchbase server status: [' Process: 10438 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-04-07 20:09:28,060] - [remote_util] [140562577667840] - INFO - Checking for process beam.smp on linux [2020-04-07 20:09:28,158] - [remote_util] [140562577667840] - INFO - process beam.smp is running on 172.23.98.165: with pid 10516 [2020-04-07 20:09:28,272] - [xdcrnewbasetests] [140562577667840] - INFO - sleep for 60 secs. ... [2020-04-07 20:10:28,332] - [xdcrnewbasetests] [140562577667840] - INFO - Updating keys @ C1 [2020-04-07 20:10:28,332] - [xdcrnewbasetests] [140562577667840] - INFO - At bucket 'default' @ C1: operation: update, key range 0 - 30000 [2020-04-07 20:10:28,388] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:10:28,511] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:10:28,620] - [xdcrnewbasetests] [140562577667840] - INFO - At bucket 'standard_bucket_1' @ C1: operation: update, key range 0 - 30000 [2020-04-07 20:10:28,691] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:10:28,912] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:10:48,612] - [xdcrnewbasetests] [140562577667840] - INFO - Batched updates loaded to cluster(s) [2020-04-07 20:10:48,631] - [xdcrnewbasetests] [140562577667840] - INFO - Deleting keys @ C1 [2020-04-07 20:10:48,631] - [xdcrnewbasetests] [140562577667840] - INFO - At bucket 'default' @ C1: operation: delete, key range 70000 - 100000 [2020-04-07 20:10:48,695] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:10:48,831] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:10:48,904] - [xdcrnewbasetests] [140562577667840] - INFO - At bucket 'standard_bucket_1' @ C1: operation: delete, key range 70000 - 100000 [2020-04-07 20:10:48,963] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:10:49,114] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:11:02,109] - [xdcrnewbasetests] [140562577667840] - INFO - Batched deletes sent to cluster(s) [2020-04-07 20:11:02,128] - [xdcrnewbasetests] [140562577667840] - INFO - sleep for 30 secs. ... [2020-04-07 20:11:32,161] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:11:32,231] - [xdcrnewbasetests] [140562577667840] - INFO - Warmed up: 100000 items on default on ip:172.23.98.17 port:8091 ssh_username:root [2020-04-07 20:11:42,262] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:11:42,387] - [xdcrnewbasetests] [140562577667840] - INFO - Warmed up: 100000 items on default on ip:172.23.98.165 port:8091 ssh_username:root [2020-04-07 20:11:52,407] - [xdcrnewbasetests] [140562577667840] - INFO - sleep for 300 secs. ... [2020-04-07 20:16:52,508] - [xdcrnewbasetests] [140562577667840] - INFO - Merging keys for replication Replication C1:default -> C2:default [2020-04-07 20:16:52,610] - [xdcrnewbasetests] [140562577667840] - INFO - src_kvstore has 70000 valid and 30000 deleted keys [2020-04-07 20:16:52,611] - [xdcrnewbasetests] [140562577667840] - INFO - dest kvstore has 0 valid and 0 deleted keys [2020-04-07 20:16:53,332] - [xdcrnewbasetests] [140562577667840] - INFO - After merging: destination bucket's kv_store now has 70000 valid keys and 0 deleted keys [2020-04-07 20:16:53,334] - [xdcrnewbasetests] [140562577667840] - INFO - Merging keys for replication Replication C1:standard_bucket_1 -> C2:standard_bucket_1 [2020-04-07 20:16:53,411] - [xdcrnewbasetests] [140562577667840] - INFO - src_kvstore has 70000 valid and 30000 deleted keys [2020-04-07 20:16:53,411] - [xdcrnewbasetests] [140562577667840] - INFO - dest kvstore has 0 valid and 0 deleted keys [2020-04-07 20:16:54,136] - [xdcrnewbasetests] [140562577667840] - INFO - After merging: destination bucket's kv_store now has 70000 valid keys and 0 deleted keys [2020-04-07 20:16:54,174] - [data_helper] [140562577667840] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.98.164', 'password': 'password', 'port': u'8091'} [2020-04-07 20:16:54,175] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:16:54,298] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.164', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2020-04-07 20:16:54,298] - [mc_bin_client] [140562577667840] - INFO - setting param: exp_pager_stime 10 [2020-04-07 20:16:54,299] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.164', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (813175214, 0, '') [2020-04-07 20:16:54,320] - [data_helper] [140562577667840] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.98.17', 'password': 'password', 'port': u'8091'} [2020-04-07 20:16:54,320] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:16:54,369] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.17', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2020-04-07 20:16:54,370] - [mc_bin_client] [140562577667840] - INFO - setting param: exp_pager_stime 10 [2020-04-07 20:16:54,370] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.17', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2682624262, 0, '') [2020-04-07 20:16:54,370] - [xdcrnewbasetests] [140562577667840] - INFO - wait for expiry pager to run on all these nodes [2020-04-07 20:16:54,400] - [data_helper] [140562577667840] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.98.164', 'password': 'password', 'port': u'8091'} [2020-04-07 20:16:54,400] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:16:54,449] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.164', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on standard_bucket_1 [2020-04-07 20:16:54,450] - [mc_bin_client] [140562577667840] - INFO - setting param: exp_pager_stime 10 [2020-04-07 20:16:54,450] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.164', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2504432781, 0, '') [2020-04-07 20:16:54,471] - [data_helper] [140562577667840] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.98.17', 'password': 'password', 'port': u'8091'} [2020-04-07 20:16:54,471] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:16:54,583] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.17', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on standard_bucket_1 [2020-04-07 20:16:54,583] - [mc_bin_client] [140562577667840] - INFO - setting param: exp_pager_stime 10 [2020-04-07 20:16:54,583] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.17', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2332678643, 0, '') [2020-04-07 20:16:54,584] - [xdcrnewbasetests] [140562577667840] - INFO - wait for expiry pager to run on all these nodes [2020-04-07 20:17:04,631] - [data_helper] [140562577667840] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.98.16', 'password': 'password', 'port': u'8091'} [2020-04-07 20:17:04,631] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:17:04,683] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.16', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2020-04-07 20:17:04,684] - [mc_bin_client] [140562577667840] - INFO - setting param: exp_pager_stime 10 [2020-04-07 20:17:04,684] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.16', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1571230914, 0, '') [2020-04-07 20:17:04,706] - [data_helper] [140562577667840] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.98.165', 'password': 'password', 'port': u'8091'} [2020-04-07 20:17:04,706] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:17:04,753] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.165', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2020-04-07 20:17:04,753] - [mc_bin_client] [140562577667840] - INFO - setting param: exp_pager_stime 10 [2020-04-07 20:17:04,754] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.165', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (936984858, 0, '') [2020-04-07 20:17:04,754] - [xdcrnewbasetests] [140562577667840] - INFO - wait for expiry pager to run on all these nodes [2020-04-07 20:17:04,782] - [data_helper] [140562577667840] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.98.16', 'password': 'password', 'port': u'8091'} [2020-04-07 20:17:04,783] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:17:04,898] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.16', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on standard_bucket_1 [2020-04-07 20:17:04,898] - [mc_bin_client] [140562577667840] - INFO - setting param: exp_pager_stime 10 [2020-04-07 20:17:04,899] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.16', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3285617177, 0, '') [2020-04-07 20:17:04,920] - [data_helper] [140562577667840] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.98.165', 'password': 'password', 'port': u'8091'} [2020-04-07 20:17:04,920] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:17:04,969] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.165', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on standard_bucket_1 [2020-04-07 20:17:04,969] - [mc_bin_client] [140562577667840] - INFO - setting param: exp_pager_stime 10 [2020-04-07 20:17:04,969] - [cluster_helper] [140562577667840] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.98.165', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (703738385, 0, '') [2020-04-07 20:17:04,969] - [xdcrnewbasetests] [140562577667840] - INFO - wait for expiry pager to run on all these nodes [2020-04-07 20:17:15,386] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:17:15,452] - [task] [140561845823232] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.98.17:8091',default bucket [2020-04-07 20:17:15,463] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:17:15,582] - [task] [140561845823232] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.98.17:8091',standard_bucket_1 bucket [2020-04-07 20:17:15,594] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:17:15,658] - [task] [140561845823232] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.98.164:8091',default bucket [2020-04-07 20:17:15,668] - [data_helper] [140561845823232] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:17:15,723] - [task] [140561845823232] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.98.164:8091',standard_bucket_1 bucket [2020-04-07 20:17:16,405] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:17:16,541] - [task] [140561871001344] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.98.165:8091',default bucket [2020-04-07 20:17:16,552] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:17:16,614] - [task] [140561871001344] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.98.165:8091',standard_bucket_1 bucket [2020-04-07 20:17:16,625] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:17:16,684] - [task] [140561871001344] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.98.16:8091',default bucket [2020-04-07 20:17:16,693] - [data_helper] [140561871001344] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:17:16,810] - [task] [140561871001344] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.98.16:8091',standard_bucket_1 bucket [2020-04-07 20:17:16,842] - [xdcrnewbasetests] [140562577667840] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.98.17 [2020-04-07 20:17:16,848] - [rest_client] [140562577667840] - INFO - http://172.23.98.17:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:17:16,883] - [xdcrnewbasetests] [140562577667840] - INFO - Current dcp queue size on C1 for default is 0 [2020-04-07 20:17:16,884] - [xdcrnewbasetests] [140562577667840] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.98.165 [2020-04-07 20:17:16,890] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:17:16,922] - [xdcrnewbasetests] [140562577667840] - INFO - Current dcp queue size on C2 for default is 0 [2020-04-07 20:17:16,922] - [xdcrnewbasetests] [140562577667840] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.98.17 [2020-04-07 20:17:16,940] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:17:16,953] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:17:21,972] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:17:21,984] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:17:27,004] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:17:27,018] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:17:32,037] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:17:32,049] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:17:37,065] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:17:37,078] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:17:42,098] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:17:42,112] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:17:47,131] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:17:47,145] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:17:52,164] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:17:52,177] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:17:57,196] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:17:57,209] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:18:02,229] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:18:02,242] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:18:07,261] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:18:07,275] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:18:12,296] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:18:12,310] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:18:17,330] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:18:17,344] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:18:22,363] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:18:22,377] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:18:27,395] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:18:27,408] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:18:32,427] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:18:32,440] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:18:37,460] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:18:37,474] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:18:42,493] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:18:42,506] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:18:47,526] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:18:47,540] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:18:52,558] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:18:52,571] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:18:57,591] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:18:57,605] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:19:02,623] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:19:02,636] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:19:07,654] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:19:07,667] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:19:12,686] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:19:12,699] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:19:17,718] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:19:17,731] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:19:22,751] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:19:22,763] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:19:27,783] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:19:27,796] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:19:32,815] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:19:32,828] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:19:37,849] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:19:37,863] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:19:42,882] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:19:42,896] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:19:47,916] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:19:47,929] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:19:52,948] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:19:52,961] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:19:57,981] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:19:57,994] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:20:03,014] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:20:03,027] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:20:08,046] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:20:08,059] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:20:13,079] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 80000 [2020-04-07 20:20:13,093] - [xdcrnewbasetests] [140562577667840] - INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 80000 [2020-04-07 20:20:18,098] - [xdcrnewbasetests] [140562577667840] - ERROR - Timeout occurs while waiting for mutations to be replicated [2020-04-07 20:20:18,098] - [xdcrnewbasetests] [140562577667840] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.98.165 [2020-04-07 20:20:18,109] - [xdcrnewbasetests] [140562577667840] - ERROR - Stat "replication_changes_left" not found [2020-04-07 20:20:18,152] - [rest_client] [140562577667840] - INFO - http://172.23.98.17:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:20:18,182] - [xdcrnewbasetests] [140562577667840] - INFO - Saw: vb_active_curr_items 70000 == 70000 expected on C1, default bucket [2020-04-07 20:20:18,220] - [rest_client] [140562577667840] - INFO - http://172.23.98.17:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute [2020-04-07 20:20:18,254] - [xdcrnewbasetests] [140562577667840] - INFO - Saw: vb_active_curr_items 70000 == 70000 expected on C1, standard_bucket_1 bucket [2020-04-07 20:20:18,291] - [rest_client] [140562577667840] - INFO - http://172.23.98.17:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:20:18,320] - [xdcrnewbasetests] [140562577667840] - INFO - Saw: vb_replica_curr_items 70000 == 70000 expected on C1, default bucket [2020-04-07 20:20:18,357] - [rest_client] [140562577667840] - INFO - http://172.23.98.17:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute [2020-04-07 20:20:18,385] - [xdcrnewbasetests] [140562577667840] - INFO - Saw: vb_replica_curr_items 70000 == 70000 expected on C1, standard_bucket_1 bucket [2020-04-07 20:20:18,427] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:20:18,460] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:20:23,465] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:20:23,499] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:20:28,504] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:20:28,541] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:20:33,546] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:20:33,581] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:20:38,585] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:20:38,617] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:20:43,622] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:20:43,655] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:20:48,660] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:20:48,697] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:20:53,703] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:20:53,739] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:20:58,739] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:20:58,772] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:21:03,778] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:21:03,814] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:21:08,819] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:21:08,855] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:21:13,860] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:21:13,898] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:21:18,904] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:21:18,936] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:21:23,942] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:21:23,977] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:21:28,982] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:21:29,021] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:21:34,026] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:21:34,060] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:21:39,066] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:21:39,098] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:21:44,103] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:21:44,136] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:21:49,141] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:21:49,174] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:21:54,180] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:21:54,215] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:21:59,220] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:21:59,260] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:22:04,265] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:22:04,301] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:22:09,307] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:22:09,339] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:22:14,344] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:22:14,377] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:22:19,382] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:22:19,415] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:22:24,421] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:22:24,454] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:22:29,459] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:22:29,493] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:22:34,498] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:22:34,531] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:22:39,532] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:22:39,564] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:22:44,569] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:22:44,602] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:22:49,607] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:22:49,639] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:22:54,645] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:22:54,678] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:22:59,684] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:22:59,717] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:23:04,722] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:23:04,754] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:23:09,759] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:23:09,794] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:23:14,799] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:23:14,837] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:23:19,842] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:23:19,878] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:23:24,883] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:23:24,921] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:23:29,926] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:23:29,965] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:23:34,970] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:23:35,005] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:23:40,010] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:23:40,042] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:23:45,047] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:23:45,087] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:23:50,093] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:23:50,130] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:23:55,135] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:23:55,169] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:24:00,171] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:24:00,204] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:24:05,209] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:24:05,242] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:24:10,247] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:24:10,282] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:24:15,287] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:24:15,320] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:24:20,325] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:24:20,358] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:24:25,363] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:24:25,395] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:24:30,401] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:24:30,434] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:24:35,439] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:24:35,472] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:24:40,477] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:24:40,512] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:24:45,514] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:24:45,550] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:24:50,555] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:24:50,589] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:24:55,594] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:24:55,628] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:00,633] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:00,664] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:05,670] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:05,703] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:10,704] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:10,738] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:15,743] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:15,780] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:20,785] - [xdcrnewbasetests] [140562577667840] - ERROR - ERROR: Timed-out waiting for active item count to match [2020-04-07 20:25:20,820] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute [2020-04-07 20:25:20,854] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_active_curr_items 85000 == 70000 expected on C2, standard_bucket_1 bucket [2020-04-07 20:25:25,854] - [xdcrnewbasetests] [140562577667840] - ERROR - ERROR: Timed-out waiting for active item count to match [2020-04-07 20:25:25,888] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:25,926] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:28,930] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:28,963] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:31,966] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:31,997] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:35,000] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:35,032] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:38,036] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:38,071] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:41,075] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:41,108] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:44,111] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:44,143] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:47,145] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:47,178] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:50,182] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:50,214] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:53,217] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:53,252] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:56,255] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:56,285] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:25:59,288] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:25:59,325] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:02,328] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:02,362] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:05,365] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:05,401] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:08,404] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:08,437] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:11,441] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:11,474] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:14,478] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:14,509] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:17,512] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:17,544] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:20,547] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:20,577] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:23,581] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:23,615] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:26,618] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:26,651] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:29,654] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:29,685] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:32,689] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:32,724] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:35,727] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:35,760] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:38,763] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:38,798] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:41,801] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:41,835] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:44,838] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:44,872] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:47,875] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:47,908] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:50,909] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:50,942] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:53,945] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:53,978] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:26:56,981] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:26:57,011] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:00,015] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:00,055] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:03,058] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:03,089] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:06,092] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:06,125] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:09,129] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:09,161] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:12,165] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:12,199] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:15,202] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:15,237] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:18,240] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:18,276] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:21,279] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:21,311] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:24,314] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:24,347] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:27,350] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:27,388] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:30,391] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:30,424] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:33,427] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:33,464] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:36,467] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:36,499] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:39,502] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:39,539] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:42,542] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:42,579] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:45,582] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:45,616] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:48,619] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:48,655] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:51,658] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:51,690] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:54,693] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:54,728] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:27:57,731] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:27:57,762] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:00,765] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:00,796] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:03,799] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:03,834] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:06,837] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:06,873] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:09,876] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:09,909] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:12,913] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:12,947] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:15,951] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:15,985] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:18,988] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:19,021] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:22,024] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:22,057] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:25,061] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:25,099] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:28,102] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:28,134] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:31,137] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:31,172] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:34,175] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:34,207] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:37,210] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:37,242] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:40,245] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:40,281] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:43,285] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:43,317] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:46,319] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:46,351] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:49,354] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:49,387] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:52,390] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:52,424] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:55,427] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:55,461] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:28:58,464] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:28:58,501] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:01,504] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:01,539] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:04,542] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:04,578] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:07,581] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:07,617] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:10,621] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:10,655] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:13,658] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:13,694] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:16,697] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:16,728] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:19,732] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:19,763] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:22,766] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:22,800] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:25,803] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:25,835] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:28,839] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:28,872] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:31,876] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:31,910] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:34,913] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:34,948] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:37,951] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:37,988] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:40,991] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:41,026] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:44,030] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:44,061] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:47,064] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:47,097] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:50,100] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:50,132] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:53,136] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:53,169] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:56,172] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:56,206] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:29:59,209] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:29:59,243] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:30:02,246] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:30:02,281] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:30:05,284] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:30:05,318] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:30:08,321] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:30:08,358] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:30:11,361] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:30:11,393] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:30:14,396] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:30:14,429] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:30:17,432] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:30:17,462] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:30:20,465] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:30:20,496] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:30:23,500] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:30:23,538] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, default bucket [2020-04-07 20:30:26,541] - [xdcrnewbasetests] [140562577667840] - ERROR - ERROR: Timed-out waiting for replica item count to match [2020-04-07 20:30:26,543] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.165 with username:root [2020-04-07 20:30:26,640] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.165 [2020-04-07 20:30:26,926] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:30:27,320] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:30:27,320] - [xdcrnewbasetests] [140562577667840] - INFO - Executing cbvdiff for bucket default [2020-04-07 20:30:27,321] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: /opt/couchbase/bin/cbvdiff -b default 172.23.98.165:11210,172.23.98.16:11210 [2020-04-07 20:30:27,499] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:30:27,499] - [remote_util] [140562577667840] - INFO - Authentication error for default [2020-04-07 20:30:27,500] - [xdcrnewbasetests] [140562577667840] - INFO - Executing cbvdiff for bucket standard_bucket_1 [2020-04-07 20:30:27,500] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: /opt/couchbase/bin/cbvdiff -b standard_bucket_1 172.23.98.165:11210,172.23.98.16:11210 [2020-04-07 20:30:27,566] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:30:27,566] - [remote_util] [140562577667840] - INFO - Authentication error for standard_bucket_1 [2020-04-07 20:30:27,709] - [rest_client] [140562577667840] - INFO - http://172.23.98.165:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute [2020-04-07 20:30:27,743] - [xdcrnewbasetests] [140562577667840] - WARNING - Not Ready: vb_replica_curr_items 85000 == 70000 expected on C2, standard_bucket_1 bucket [2020-04-07 20:30:30,747] - [xdcrnewbasetests] [140562577667840] - ERROR - ERROR: Timed-out waiting for replica item count to match [2020-04-07 20:30:30,748] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.165 with username:root [2020-04-07 20:30:30,848] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.165 [2020-04-07 20:30:31,089] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:30:31,439] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:30:31,439] - [xdcrnewbasetests] [140562577667840] - INFO - Executing cbvdiff for bucket default [2020-04-07 20:30:31,439] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: /opt/couchbase/bin/cbvdiff -b default 172.23.98.165:11210,172.23.98.16:11210 [2020-04-07 20:30:31,568] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:30:31,568] - [remote_util] [140562577667840] - INFO - Authentication error for default [2020-04-07 20:30:31,568] - [xdcrnewbasetests] [140562577667840] - INFO - Executing cbvdiff for bucket standard_bucket_1 [2020-04-07 20:30:31,568] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: /opt/couchbase/bin/cbvdiff -b standard_bucket_1 172.23.98.165:11210,172.23.98.16:11210 [2020-04-07 20:30:31,697] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:30:31,697] - [remote_util] [140562577667840] - INFO - Authentication error for standard_bucket_1 [2020-04-07 20:30:31,865] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:30:31,931] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:30:32,190] - [task] [140562577667840] - INFO - 100000 items will be verified on default bucket [2020-04-07 20:30:32,248] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:30:32,315] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:30:32,512] - [task] [140562577667840] - INFO - 100000 items will be verified on standard_bucket_1 bucket [2020-04-07 20:30:32,572] - [task] [140562569275136] - INFO - 0 items were verified [2020-04-07 20:30:32,573] - [task] [140561854215936] - INFO - 0 items were verified [2020-04-07 20:30:40,005] - [task] [140561854215936] - INFO - 10000 items were verified [2020-04-07 20:30:40,046] - [task] [140562569275136] - INFO - 10000 items were verified [2020-04-07 20:30:47,609] - [task] [140561854215936] - INFO - 20000 items were verified [2020-04-07 20:30:47,765] - [task] [140562569275136] - INFO - 20000 items were verified [2020-04-07 20:30:55,542] - [task] [140561854215936] - INFO - 30000 items were verified [2020-04-07 20:30:55,697] - [task] [140562569275136] - INFO - 30000 items were verified [2020-04-07 20:31:03,227] - [task] [140561854215936] - INFO - 40000 items were verified [2020-04-07 20:31:03,604] - [task] [140562569275136] - INFO - 40000 items were verified [2020-04-07 20:31:10,918] - [task] [140561854215936] - INFO - 50000 items were verified [2020-04-07 20:31:11,097] - [task] [140562569275136] - INFO - 50000 items were verified [2020-04-07 20:31:18,518] - [task] [140562569275136] - INFO - 60000 items were verified [2020-04-07 20:31:18,686] - [task] [140561854215936] - INFO - 60000 items were verified [2020-04-07 20:31:26,283] - [task] [140562569275136] - INFO - 70000 items were verified [2020-04-07 20:31:26,401] - [task] [140561854215936] - INFO - 70000 items were verified [2020-04-07 20:31:28,907] - [task] [140562569275136] - INFO - 80000 items were verified [2020-04-07 20:31:28,958] - [task] [140561854215936] - INFO - 80000 items were verified [2020-04-07 20:31:31,519] - [task] [140562569275136] - INFO - 90000 items were verified [2020-04-07 20:31:31,550] - [task] [140561854215936] - INFO - 90000 items were verified [2020-04-07 20:31:34,072] - [task] [140562569275136] - INFO - 100000 items were verified [2020-04-07 20:31:34,072] - [task] [140562569275136] - INFO - 100000 items were verified in 61.8812839985 sec.the average number of ops - 1615.99744011 per second [2020-04-07 20:31:34,105] - [task] [140561854215936] - INFO - 100000 items were verified [2020-04-07 20:31:34,105] - [task] [140561854215936] - INFO - 100000 items were verified in 61.5926201344 sec.the average number of ops - 1623.57108181 per second [2020-04-07 20:31:34,284] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:31:34,354] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:31:34,537] - [task] [140562577667840] - INFO - 70000 items will be verified on default bucket [2020-04-07 20:31:34,697] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:31:34,718] - [task] [140561854215936] - INFO - 0 items were verified [2020-04-07 20:31:34,862] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:31:35,148] - [task] [140562577667840] - INFO - 70000 items will be verified on standard_bucket_1 bucket [2020-04-07 20:31:35,150] - [xdcrnewbasetests] [140562577667840] - ERROR - Key: C1-key-11552 Bad hash result: 23827 != 18476 [2020-04-07 20:31:35,150] - [xdcrnewbasetests] [140562577667840] - INFO - Verifying RevIds for Couchbase Cluster: C1, Master Ip: 172.23.98.17 -> Couchbase Cluster: C2, Master Ip: 172.23.98.165, bucket default [2020-04-07 20:31:35,252] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:31:35,484] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:31:35,652] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 default [2020-04-07 20:31:35,768] - [task] [140561854215936] - INFO - 0 items were verified [2020-04-07 20:31:35,924] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 default [2020-04-07 20:31:36,086] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.16:11210 default [2020-04-07 20:31:36,275] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.165:11210 default [2020-04-07 20:31:36,778] - [task] [140562577667840] - INFO - RevID verification: in progress for default ... [2020-04-07 20:31:36,779] - [xdcrnewbasetests] [140562577667840] - INFO - Verifying RevIds for Couchbase Cluster: C1, Master Ip: 172.23.98.17 -> Couchbase Cluster: C2, Master Ip: 172.23.98.165, bucket standard_bucket_1 [2020-04-07 20:31:36,840] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:31:36,986] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:31:37,119] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.164:11210 standard_bucket_1 [2020-04-07 20:31:37,185] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.17:11210 standard_bucket_1 [2020-04-07 20:31:37,407] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.16:11210 standard_bucket_1 [2020-04-07 20:31:37,478] - [data_helper] [140562577667840] - INFO - creating direct client 172.23.98.165:11210 standard_bucket_1 [2020-04-07 20:31:37,587] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-13098, bucket:default ===== [2020-04-07 20:31:37,587] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:1 [2020-04-07 20:31:37,588] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315436211044352, Destination cas:1586315323268399104, Error Count:2 [2020-04-07 20:31:37,588] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315436211044352, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,588] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323268399104, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,609] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-19161, bucket:default ===== [2020-04-07 20:31:37,609] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:3 [2020-04-07 20:31:37,610] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315440359800832, Destination cas:1586315324320710656, Error Count:4 [2020-04-07 20:31:37,610] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315440359800832, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,610] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324320710656, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,617] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-22689, bucket:default ===== [2020-04-07 20:31:37,617] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:5 [2020-04-07 20:31:37,618] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315442615681024, Destination cas:1586315324839034880, Error Count:6 [2020-04-07 20:31:37,618] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315442615681024, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,618] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324839034880, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,627] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-28008, bucket:default ===== [2020-04-07 20:31:37,628] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:7 [2020-04-07 20:31:37,628] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315446573727744, Destination cas:1586315321512755200, Error Count:8 [2020-04-07 20:31:37,629] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315446573727744, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,629] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321512755200, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,638] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-1669, bucket:default ===== [2020-04-07 20:31:37,639] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:9 [2020-04-07 20:31:37,640] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429223464960, Destination cas:1586315321115344896, Error Count:10 [2020-04-07 20:31:37,641] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429223464960, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,641] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321115344896, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,659] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-23818, bucket:default ===== [2020-04-07 20:31:37,659] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:11 [2020-04-07 20:31:37,659] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315443247382528, Destination cas:1586315325028499456, Error Count:12 [2020-04-07 20:31:37,660] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315443247382528, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,660] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325028499456, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,662] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-2696, bucket:default ===== [2020-04-07 20:31:37,663] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:13 [2020-04-07 20:31:37,664] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429409390592, Destination cas:1586315321291833344, Error Count:14 [2020-04-07 20:31:37,664] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429409390592, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,664] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321291833344, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,669] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-20044, bucket:default ===== [2020-04-07 20:31:37,671] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:15 [2020-04-07 20:31:37,671] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315441137713152, Destination cas:1586315324496871424, Error Count:16 [2020-04-07 20:31:37,671] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441137713152, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,671] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324496871424, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,678] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-21572, bucket:default ===== [2020-04-07 20:31:37,678] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:17 [2020-04-07 20:31:37,678] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315441922310144, Destination cas:1586315324668706816, Error Count:18 [2020-04-07 20:31:37,679] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441922310144, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,680] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324668706816, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,702] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-11257, bucket:default ===== [2020-04-07 20:31:37,703] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:19 [2020-04-07 20:31:37,703] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315434576576512, Destination cas:1586315322912276480, Error Count:20 [2020-04-07 20:31:37,704] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315434576576512, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,704] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322912276480, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,714] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-21688, bucket:default ===== [2020-04-07 20:31:37,714] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:21 [2020-04-07 20:31:37,715] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315441929519104, Destination cas:1586315324670672896, Error Count:22 [2020-04-07 20:31:37,715] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441929519104, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,715] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324670672896, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,720] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-3866, bucket:default ===== [2020-04-07 20:31:37,721] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:23 [2020-04-07 20:31:37,721] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429588828160, Destination cas:1586315321471008768, Error Count:24 [2020-04-07 20:31:37,722] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429588828160, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,722] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321471008768, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,725] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-26943, bucket:default ===== [2020-04-07 20:31:37,725] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:25 [2020-04-07 20:31:37,726] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315445287387136, Destination cas:1586315321155649536, Error Count:26 [2020-04-07 20:31:37,726] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445287387136, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,726] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321155649536, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,729] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-14422, bucket:default ===== [2020-04-07 20:31:37,729] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:27 [2020-04-07 20:31:37,729] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315437025984512, Destination cas:1586315323444822016, Error Count:28 [2020-04-07 20:31:37,729] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315437025984512, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,729] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323444822016, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,732] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-8590, bucket:default ===== [2020-04-07 20:31:37,733] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:29 [2020-04-07 20:31:37,733] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315432432435200, Destination cas:1586315322393427968, Error Count:30 [2020-04-07 20:31:37,733] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315432432435200, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,733] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322393427968, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,741] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-17501, bucket:default ===== [2020-04-07 20:31:37,741] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:31 [2020-04-07 20:31:37,741] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315439084142592, Destination cas:1586315323964784640, Error Count:32 [2020-04-07 20:31:37,741] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315439084142592, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,742] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323964784640, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,747] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-25668, bucket:default ===== [2020-04-07 20:31:37,747] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:33 [2020-04-07 20:31:37,748] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315444617478144, Destination cas:1586315320957206528, Error Count:34 [2020-04-07 20:31:37,748] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315444617478144, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,748] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320957206528, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,752] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-9261, bucket:default ===== [2020-04-07 20:31:37,753] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:35 [2020-04-07 20:31:37,754] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315433065447424, Destination cas:1586315322558971904, Error Count:36 [2020-04-07 20:31:37,754] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315433065447424, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,754] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322558971904, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,768] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-28087, bucket:default ===== [2020-04-07 20:31:37,768] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:37 [2020-04-07 20:31:37,769] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315446570844160, Destination cas:1586315321508823040, Error Count:38 [2020-04-07 20:31:37,769] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315446570844160, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,770] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321508823040, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,774] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-15104, bucket:default ===== [2020-04-07 20:31:37,774] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:39 [2020-04-07 20:31:37,774] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315437837385728, Destination cas:1586315323614887936, Error Count:40 [2020-04-07 20:31:37,774] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315437837385728, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,775] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323614887936, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,778] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-23307, bucket:default ===== [2020-04-07 20:31:37,778] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:41 [2020-04-07 20:31:37,778] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315443240697856, Destination cas:1586315325025288192, Error Count:42 [2020-04-07 20:31:37,779] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315443240697856, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,779] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325025288192, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,796] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-3366, bucket:default ===== [2020-04-07 20:31:37,797] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:43 [2020-04-07 20:31:37,797] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429594529792, Destination cas:1586315321483395072, Error Count:44 [2020-04-07 20:31:37,797] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429594529792, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,797] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321483395072, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,805] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-24116, bucket:default ===== [2020-04-07 20:31:37,805] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:45 [2020-04-07 20:31:37,805] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315443994165248, Destination cas:1586315325194502144, Error Count:46 [2020-04-07 20:31:37,806] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315443994165248, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,806] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325194502144, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,812] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-28977, bucket:default ===== [2020-04-07 20:31:37,812] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:47 [2020-04-07 20:31:37,812] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315446566912000, Destination cas:1586315321499058176, Error Count:48 [2020-04-07 20:31:37,813] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315446566912000, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,813] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321499058176, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,820] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-9455, bucket:default ===== [2020-04-07 20:31:37,821] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:49 [2020-04-07 20:31:37,821] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315433069314048, Destination cas:1586315322561396736, Error Count:50 [2020-04-07 20:31:37,821] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315433069314048, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,821] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322561396736, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,833] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16352, bucket:default ===== [2020-04-07 20:31:37,833] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:51 [2020-04-07 20:31:37,834] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315438486978560, Destination cas:1586315323788427264, Error Count:52 [2020-04-07 20:31:37,834] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438486978560, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,835] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323788427264, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,849] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-8641, bucket:default ===== [2020-04-07 20:31:37,849] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:53 [2020-04-07 20:31:37,850] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315432430600192, Destination cas:1586315322392313856, Error Count:54 [2020-04-07 20:31:37,851] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315432430600192, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,851] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322392313856, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,856] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16963, bucket:default ===== [2020-04-07 20:31:37,857] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:55 [2020-04-07 20:31:37,857] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315438503428096, Destination cas:1586315323799502848, Error Count:56 [2020-04-07 20:31:37,857] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438503428096, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,858] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323799502848, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,860] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-11083, bucket:default ===== [2020-04-07 20:31:37,860] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:57 [2020-04-07 20:31:37,861] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315434583326720, Destination cas:1586315322915487744, Error Count:58 [2020-04-07 20:31:37,861] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315434583326720, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,861] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322915487744, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,875] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-484, bucket:default ===== [2020-04-07 20:31:37,875] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:59 [2020-04-07 20:31:37,875] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315428949852160, Destination cas:1586315320931581952, Error Count:60 [2020-04-07 20:31:37,875] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315428949852160, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,876] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320931581952, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,877] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-9377, bucket:default ===== [2020-04-07 20:31:37,877] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:61 [2020-04-07 20:31:37,877] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315433082224640, Destination cas:1586315322567229440, Error Count:62 [2020-04-07 20:31:37,877] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315433082224640, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,878] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322567229440, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,884] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-20395, bucket:default ===== [2020-04-07 20:31:37,884] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:63 [2020-04-07 20:31:37,884] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315441131356160, Destination cas:1586315324492677120, Error Count:64 [2020-04-07 20:31:37,884] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441131356160, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,884] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324492677120, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,886] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-25433, bucket:default ===== [2020-04-07 20:31:37,886] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:65 [2020-04-07 20:31:37,886] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315444620820480, Destination cas:1586315320960679936, Error Count:66 [2020-04-07 20:31:37,886] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315444620820480, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,887] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320960679936, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,894] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16317, bucket:default ===== [2020-04-07 20:31:37,894] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:67 [2020-04-07 20:31:37,894] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315438503231488, Destination cas:1586315323799371776, Error Count:68 [2020-04-07 20:31:37,895] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438503231488, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,895] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323799371776, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,903] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-15775, bucket:default ===== [2020-04-07 20:31:37,903] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:69 [2020-04-07 20:31:37,903] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315437834567680, Destination cas:1586315323613446144, Error Count:70 [2020-04-07 20:31:37,904] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315437834567680, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,905] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323613446144, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,907] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-18248, bucket:default ===== [2020-04-07 20:31:37,907] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:71 [2020-04-07 20:31:37,907] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315439686352896, Destination cas:1586315324150054912, Error Count:72 [2020-04-07 20:31:37,907] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315439686352896, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,908] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324150054912, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,913] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-4507, bucket:default ===== [2020-04-07 20:31:37,913] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:73 [2020-04-07 20:31:37,913] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429760466944, Destination cas:1586315321648021504, Error Count:74 [2020-04-07 20:31:37,914] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429760466944, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,914] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321648021504, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,916] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-5363, bucket:default ===== [2020-04-07 20:31:37,916] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:75 [2020-04-07 20:31:37,916] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315430199230464, Destination cas:1586315321839452160, Error Count:76 [2020-04-07 20:31:37,917] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315430199230464, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,917] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321839452160, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,920] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-12002, bucket:default ===== [2020-04-07 20:31:37,920] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:77 [2020-04-07 20:31:37,920] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315435365367808, Destination cas:1586315323092762624, Error Count:78 [2020-04-07 20:31:37,921] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315435365367808, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,921] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323092762624, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,924] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-2979, bucket:default ===== [2020-04-07 20:31:37,924] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:79 [2020-04-07 20:31:37,924] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429410963456, Destination cas:1586315321294127104, Error Count:80 [2020-04-07 20:31:37,924] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429410963456, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,924] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321294127104, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,932] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-27599, bucket:default ===== [2020-04-07 20:31:37,932] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:81 [2020-04-07 20:31:37,933] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315445912403968, Destination cas:1586315321325780992, Error Count:82 [2020-04-07 20:31:37,933] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445912403968, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,934] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321325780992, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,938] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-465, bucket:default ===== [2020-04-07 20:31:37,938] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:83 [2020-04-07 20:31:37,939] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315428960731136, Destination cas:1586315320935776256, Error Count:84 [2020-04-07 20:31:37,939] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315428960731136, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,939] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320935776256, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,944] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-29471, bucket:default ===== [2020-04-07 20:31:37,945] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:85 [2020-04-07 20:31:37,945] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315447221223424, Destination cas:1586315321693044736, Error Count:86 [2020-04-07 20:31:37,945] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447221223424, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,945] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321693044736, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,947] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-6102, bucket:default ===== [2020-04-07 20:31:37,947] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:87 [2020-04-07 20:31:37,947] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315430932774912, Destination cas:1586315322000867328, Error Count:88 [2020-04-07 20:31:37,948] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315430932774912, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,948] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322000867328, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,951] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16244, bucket:default ===== [2020-04-07 20:31:37,951] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:89 [2020-04-07 20:31:37,951] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315438498578432, Destination cas:1586315323797012480, Error Count:90 [2020-04-07 20:31:37,951] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438498578432, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,952] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323797012480, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,954] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-3211, bucket:default ===== [2020-04-07 20:31:37,955] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:91 [2020-04-07 20:31:37,955] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429586534400, Destination cas:1586315321469435904, Error Count:92 [2020-04-07 20:31:37,955] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429586534400, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,956] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321469435904, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,972] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-11552, bucket:default ===== [2020-04-07 20:31:37,972] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:93 [2020-04-07 20:31:37,973] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315434577362944, Destination cas:1586315322912538624, Error Count:94 [2020-04-07 20:31:37,973] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315434577362944, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,973] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322912538624, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,985] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-15179, bucket:default ===== [2020-04-07 20:31:37,985] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:95 [2020-04-07 20:31:37,986] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315437833846784, Destination cas:1586315323613184000, Error Count:96 [2020-04-07 20:31:37,986] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315437833846784, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,986] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323613184000, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,995] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-8253, bucket:default ===== [2020-04-07 20:31:37,995] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:97 [2020-04-07 20:31:37,996] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315432420638720, Destination cas:1586315322377568256, Error Count:98 [2020-04-07 20:31:37,996] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315432420638720, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:37,996] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322377568256, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,010] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-2354, bucket:default ===== [2020-04-07 20:31:38,011] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:99 [2020-04-07 20:31:38,011] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429411422208, Destination cas:1586315321294913536, Error Count:100 [2020-04-07 20:31:38,011] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429411422208, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,011] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321294913536, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,024] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-21135, bucket:default ===== [2020-04-07 20:31:38,024] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:101 [2020-04-07 20:31:38,025] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315441926569984, Destination cas:1586315324669886464, Error Count:102 [2020-04-07 20:31:38,025] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441926569984, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,026] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324669886464, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,027] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-19172, bucket:default ===== [2020-04-07 20:31:38,027] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:103 [2020-04-07 20:31:38,028] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315440369238016, Destination cas:1586315324323594240, Error Count:104 [2020-04-07 20:31:38,028] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315440369238016, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,028] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324323594240, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,030] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-13524, bucket:default ===== [2020-04-07 20:31:38,030] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:105 [2020-04-07 20:31:38,031] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315436213796864, Destination cas:1586315323269644288, Error Count:106 [2020-04-07 20:31:38,031] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315436213796864, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,031] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323269644288, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,039] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-23662, bucket:default ===== [2020-04-07 20:31:38,039] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:107 [2020-04-07 20:31:38,039] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315443239387136, Destination cas:1586315325024763904, Error Count:108 [2020-04-07 20:31:38,039] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315443239387136, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,040] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325024763904, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,045] - [task] [140562577667840] - INFO - RevID verification: in progress for standard_bucket_1 ... [2020-04-07 20:31:38,048] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-21633, bucket:default ===== [2020-04-07 20:31:38,048] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:109 [2020-04-07 20:31:38,048] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315441929191424, Destination cas:1586315324670541824, Error Count:110 [2020-04-07 20:31:38,048] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441929191424, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,048] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324670541824, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,069] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-12723, bucket:default ===== [2020-04-07 20:31:38,069] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:111 [2020-04-07 20:31:38,069] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315435370020864, Destination cas:1586315323096301568, Error Count:112 [2020-04-07 20:31:38,069] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315435370020864, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,069] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323096301568, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,074] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-22218, bucket:default ===== [2020-04-07 20:31:38,074] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:113 [2020-04-07 20:31:38,074] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315442626625536, Destination cas:1586315324843229184, Error Count:114 [2020-04-07 20:31:38,074] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315442626625536, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,074] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324843229184, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,078] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16351, bucket:default ===== [2020-04-07 20:31:38,078] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:115 [2020-04-07 20:31:38,079] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315438503559168, Destination cas:1586315323799568384, Error Count:116 [2020-04-07 20:31:38,079] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438503559168, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,079] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323799568384, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,082] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-851, bucket:default ===== [2020-04-07 20:31:38,082] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:117 [2020-04-07 20:31:38,082] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315428949590016, Destination cas:1586315320931254272, Error Count:118 [2020-04-07 20:31:38,082] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315428949590016, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,082] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320931254272, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,087] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-25244, bucket:default ===== [2020-04-07 20:31:38,087] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:119 [2020-04-07 20:31:38,087] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315444631764992, Destination cas:1586315320977260544, Error Count:120 [2020-04-07 20:31:38,087] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315444631764992, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,087] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320977260544, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,090] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-26962, bucket:default ===== [2020-04-07 20:31:38,090] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:121 [2020-04-07 20:31:38,090] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315445294006272, Destination cas:1586315321158533120, Error Count:122 [2020-04-07 20:31:38,090] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445294006272, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,090] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321158533120, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,092] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-2481, bucket:default ===== [2020-04-07 20:31:38,093] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:123 [2020-04-07 20:31:38,093] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429408473088, Destination cas:1586315321291374592, Error Count:124 [2020-04-07 20:31:38,093] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429408473088, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,093] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321291374592, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,096] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-4677, bucket:default ===== [2020-04-07 20:31:38,096] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:125 [2020-04-07 20:31:38,096] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429762039808, Destination cas:1586315321652477952, Error Count:126 [2020-04-07 20:31:38,096] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429762039808, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,096] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321652477952, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,102] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-807, bucket:default ===== [2020-04-07 20:31:38,102] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:127 [2020-04-07 20:31:38,102] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315428950704128, Destination cas:1586315320932564992, Error Count:128 [2020-04-07 20:31:38,102] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315428950704128, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,102] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320932564992, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,104] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-3631, bucket:default ===== [2020-04-07 20:31:38,104] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:129 [2020-04-07 20:31:38,104] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429583585280, Destination cas:1586315321466945536, Error Count:130 [2020-04-07 20:31:38,104] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429583585280, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,104] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321466945536, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,116] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-26108, bucket:default ===== [2020-04-07 20:31:38,116] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:131 [2020-04-07 20:31:38,116] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315445284700160, Destination cas:1586315321152110592, Error Count:132 [2020-04-07 20:31:38,116] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445284700160, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,116] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321152110592, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,117] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-24441, bucket:default ===== [2020-04-07 20:31:38,117] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:133 [2020-04-07 20:31:38,118] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315443986759680, Destination cas:1586315325191749632, Error Count:134 [2020-04-07 20:31:38,118] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315443986759680, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,118] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325191749632, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,124] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-29154, bucket:default ===== [2020-04-07 20:31:38,124] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:135 [2020-04-07 20:31:38,124] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315447214145536, Destination cas:1586315321690685440, Error Count:136 [2020-04-07 20:31:38,125] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447214145536, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,125] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321690685440, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,130] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-1234, bucket:default ===== [2020-04-07 20:31:38,130] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:137 [2020-04-07 20:31:38,130] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429219139584, Destination cas:1586315321106497536, Error Count:138 [2020-04-07 20:31:38,131] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429219139584, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,131] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321106497536, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,132] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-17049, bucket:default ===== [2020-04-07 20:31:38,132] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:139 [2020-04-07 20:31:38,132] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315439087157248, Destination cas:1586315323966357504, Error Count:140 [2020-04-07 20:31:38,132] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315439087157248, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,132] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323966357504, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,136] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-17482, bucket:default ===== [2020-04-07 20:31:38,136] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:141 [2020-04-07 20:31:38,136] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315439102689280, Destination cas:1586315323979595776, Error Count:142 [2020-04-07 20:31:38,136] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315439102689280, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,136] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323979595776, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,139] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-21553, bucket:default ===== [2020-04-07 20:31:38,139] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:143 [2020-04-07 20:31:38,139] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315441912807424, Destination cas:1586315324657696768, Error Count:144 [2020-04-07 20:31:38,139] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441912807424, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,140] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324657696768, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,145] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-12299, bucket:default ===== [2020-04-07 20:31:38,145] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:145 [2020-04-07 20:31:38,146] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315435362615296, Destination cas:1586315323090010112, Error Count:146 [2020-04-07 20:31:38,146] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315435362615296, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,146] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323090010112, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,147] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-26166, bucket:default ===== [2020-04-07 20:31:38,147] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:147 [2020-04-07 20:31:38,147] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315445293023232, Destination cas:1586315321158008832, Error Count:148 [2020-04-07 20:31:38,147] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445293023232, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,147] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321158008832, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,151] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-15474, bucket:default ===== [2020-04-07 20:31:38,151] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:149 [2020-04-07 20:31:38,151] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315437824475136, Destination cas:1586315323608727552, Error Count:150 [2020-04-07 20:31:38,151] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315437824475136, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,151] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323608727552, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,152] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-13060, bucket:default ===== [2020-04-07 20:31:38,152] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:151 [2020-04-07 20:31:38,152] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315436213600256, Destination cas:1586315323269447680, Error Count:152 [2020-04-07 20:31:38,152] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315436213600256, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,152] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323269447680, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,156] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-12485, bucket:default ===== [2020-04-07 20:31:38,156] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:153 [2020-04-07 20:31:38,156] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315435360387072, Destination cas:1586315323088306176, Error Count:154 [2020-04-07 20:31:38,156] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315435360387072, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,156] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323088306176, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,161] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-11969, bucket:default ===== [2020-04-07 20:31:38,161] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:155 [2020-04-07 20:31:38,161] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315434570285056, Destination cas:1586315322908475392, Error Count:156 [2020-04-07 20:31:38,161] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315434570285056, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,161] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322908475392, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,166] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-22575, bucket:default ===== [2020-04-07 20:31:38,166] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:157 [2020-04-07 20:31:38,166] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315442620071936, Destination cas:1586315324840738816, Error Count:158 [2020-04-07 20:31:38,166] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315442620071936, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,166] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324840738816, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,174] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16759, bucket:default ===== [2020-04-07 20:31:38,174] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:159 [2020-04-07 20:31:38,174] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315438498185216, Destination cas:1586315323796750336, Error Count:160 [2020-04-07 20:31:38,175] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438498185216, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,175] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323796750336, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,176] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-4233, bucket:default ===== [2020-04-07 20:31:38,177] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:161 [2020-04-07 20:31:38,177] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429766955008, Destination cas:1586315321655296000, Error Count:162 [2020-04-07 20:31:38,177] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429766955008, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,177] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321655296000, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,184] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-7974, bucket:default ===== [2020-04-07 20:31:38,184] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:163 [2020-04-07 20:31:38,184] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315431788281856, Destination cas:1586315322210385920, Error Count:164 [2020-04-07 20:31:38,184] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315431788281856, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,184] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322210385920, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,185] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-26868, bucket:default ===== [2020-04-07 20:31:38,185] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:165 [2020-04-07 20:31:38,185] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315445291122688, Destination cas:1586315321157156864, Error Count:166 [2020-04-07 20:31:38,185] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445291122688, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,185] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321157156864, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,186] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-17597, bucket:default ===== [2020-04-07 20:31:38,186] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:167 [2020-04-07 20:31:38,187] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315439094890496, Destination cas:1586315323971731456, Error Count:168 [2020-04-07 20:31:38,187] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315439094890496, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,187] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323971731456, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,188] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-15986, bucket:default ===== [2020-04-07 20:31:38,188] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:169 [2020-04-07 20:31:38,189] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315437832929280, Destination cas:1586315323612921856, Error Count:170 [2020-04-07 20:31:38,189] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315437832929280, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,189] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323612921856, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,192] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16344, bucket:default ===== [2020-04-07 20:31:38,192] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:171 [2020-04-07 20:31:38,192] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315438493401088, Destination cas:1586315323793408000, Error Count:172 [2020-04-07 20:31:38,192] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438493401088, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,192] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323793408000, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,195] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-29216, bucket:default ===== [2020-04-07 20:31:38,195] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:173 [2020-04-07 20:31:38,195] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315447223320576, Destination cas:1586315321693896704, Error Count:174 [2020-04-07 20:31:38,195] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447223320576, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,195] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321693896704, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,196] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-23146, bucket:default ===== [2020-04-07 20:31:38,197] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:175 [2020-04-07 20:31:38,197] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315443246530560, Destination cas:1586315325027254272, Error Count:176 [2020-04-07 20:31:38,197] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315443246530560, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,197] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325027254272, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,200] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-19504, bucket:default ===== [2020-04-07 20:31:38,200] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:177 [2020-04-07 20:31:38,200] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315440375005184, Destination cas:1586315324327198720, Error Count:178 [2020-04-07 20:31:38,201] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315440375005184, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,201] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324327198720, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,203] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-26428, bucket:default ===== [2020-04-07 20:31:38,203] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:179 [2020-04-07 20:31:38,203] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315445285355520, Destination cas:1586315321152569344, Error Count:180 [2020-04-07 20:31:38,203] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445285355520, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,203] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321152569344, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,205] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16442, bucket:default ===== [2020-04-07 20:31:38,205] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:181 [2020-04-07 20:31:38,205] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315438501199872, Destination cas:1586315323798192128, Error Count:182 [2020-04-07 20:31:38,206] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438501199872, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,206] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323798192128, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,207] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-7148, bucket:default ===== [2020-04-07 20:31:38,207] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:183 [2020-04-07 20:31:38,207] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315431794704384, Destination cas:1586315322212155392, Error Count:184 [2020-04-07 20:31:38,207] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315431794704384, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,207] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322212155392, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,219] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-4021, bucket:default ===== [2020-04-07 20:31:38,219] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:185 [2020-04-07 20:31:38,219] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315429767151616, Destination cas:1586315321655361536, Error Count:186 [2020-04-07 20:31:38,219] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429767151616, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,220] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321655361536, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,225] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-13691, bucket:default ===== [2020-04-07 20:31:38,225] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:187 [2020-04-07 20:31:38,225] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315436213600256, Destination cas:1586315323269447680, Error Count:188 [2020-04-07 20:31:38,225] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315436213600256, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,225] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323269447680, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,228] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-13515, bucket:default ===== [2020-04-07 20:31:38,228] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:189 [2020-04-07 20:31:38,228] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315436209274880, Destination cas:1586315323267678208, Error Count:190 [2020-04-07 20:31:38,228] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315436209274880, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,228] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323267678208, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,231] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-11836, bucket:default ===== [2020-04-07 20:31:38,231] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:191 [2020-04-07 20:31:38,231] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315434583523328, Destination cas:1586315322915618816, Error Count:192 [2020-04-07 20:31:38,231] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315434583523328, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,231] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322915618816, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,242] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-23162, bucket:default ===== [2020-04-07 20:31:38,242] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:193 [2020-04-07 20:31:38,242] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315443246661632, Destination cas:1586315325027450880, Error Count:194 [2020-04-07 20:31:38,242] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315443246661632, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,242] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325027450880, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,245] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-10168, bucket:default ===== [2020-04-07 20:31:38,245] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:195 [2020-04-07 20:31:38,245] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315433738567680, Destination cas:1586315322730086400, Error Count:196 [2020-04-07 20:31:38,245] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315433738567680, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,245] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322730086400, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,251] - [task] [140561854215936] - ERROR - ===== Verifying rev_ids failed for key: C1-key-21635, bucket:default ===== [2020-04-07 20:31:38,252] - [task] [140561854215936] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:197 [2020-04-07 20:31:38,252] - [task] [140561854215936] - ERROR - cas mismatch: Source cas:1586315441928994816, Destination cas:1586315324670541824, Error Count:198 [2020-04-07 20:31:38,252] - [task] [140561854215936] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441928994816, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,252] - [task] [140561854215936] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324670541824, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,588] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-13098, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,589] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:1 [2020-04-07 20:31:38,589] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315439417982976, Destination cas:1586315323301691392, Error Count:2 [2020-04-07 20:31:38,589] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315439417982976, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,589] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323301691392, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,598] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-19161, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,599] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:3 [2020-04-07 20:31:38,599] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315443905232896, Destination cas:1586315324368224256, Error Count:4 [2020-04-07 20:31:38,599] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315443905232896, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,599] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324368224256, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,604] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-22689, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,604] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:5 [2020-04-07 20:31:38,604] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315445915156480, Destination cas:1586315324886155264, Error Count:6 [2020-04-07 20:31:38,605] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445915156480, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,605] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324886155264, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,608] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-28008, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,609] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:7 [2020-04-07 20:31:38,609] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315448303288320, Destination cas:1586315321548668928, Error Count:8 [2020-04-07 20:31:38,609] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315448303288320, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,609] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321548668928, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,612] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-1669, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,612] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:9 [2020-04-07 20:31:38,613] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315430589300736, Destination cas:1586315321110167552, Error Count:10 [2020-04-07 20:31:38,613] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315430589300736, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,613] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321110167552, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,622] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-23818, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,622] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:11 [2020-04-07 20:31:38,622] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315446563504128, Destination cas:1586315325071949824, Error Count:12 [2020-04-07 20:31:38,622] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315446563504128, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,623] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325071949824, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,624] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-2696, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,624] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:13 [2020-04-07 20:31:38,624] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315431344865280, Destination cas:1586315321284952064, Error Count:14 [2020-04-07 20:31:38,624] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315431344865280, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,624] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321284952064, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,627] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-20044, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,628] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:15 [2020-04-07 20:31:38,628] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315444591460352, Destination cas:1586315324549890048, Error Count:16 [2020-04-07 20:31:38,628] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315444591460352, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,628] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324549890048, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,631] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-21572, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,631] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:17 [2020-04-07 20:31:38,631] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315445272379392, Destination cas:1586315324722053120, Error Count:18 [2020-04-07 20:31:38,632] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445272379392, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,632] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324722053120, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,640] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-11257, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,640] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:19 [2020-04-07 20:31:38,641] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315438174306304, Destination cas:1586315322938425344, Error Count:20 [2020-04-07 20:31:38,641] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438174306304, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,641] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322938425344, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,647] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-21688, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,647] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:21 [2020-04-07 20:31:38,647] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315445278081024, Destination cas:1586315324723953664, Error Count:22 [2020-04-07 20:31:38,647] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445278081024, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,647] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324723953664, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,650] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-3866, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,650] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:23 [2020-04-07 20:31:38,651] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315432036139008, Destination cas:1586315321460654080, Error Count:24 [2020-04-07 20:31:38,651] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315432036139008, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,651] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321460654080, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,653] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-26943, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,653] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:25 [2020-04-07 20:31:38,653] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315447942905856, Destination cas:1586315321169412096, Error Count:26 [2020-04-07 20:31:38,653] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447942905856, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,654] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321169412096, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,655] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-14422, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,655] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:27 [2020-04-07 20:31:38,655] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315440026484736, Destination cas:1586315323480145920, Error Count:28 [2020-04-07 20:31:38,655] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315440026484736, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,656] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323480145920, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,657] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-8590, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,658] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:29 [2020-04-07 20:31:38,658] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315435686756352, Destination cas:1586315322405421056, Error Count:30 [2020-04-07 20:31:38,658] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315435686756352, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,658] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322405421056, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,662] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-17501, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,662] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:31 [2020-04-07 20:31:38,662] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315442440765440, Destination cas:1586315323996241920, Error Count:32 [2020-04-07 20:31:38,663] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315442440765440, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,663] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323996241920, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,666] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-25668, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,666] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:33 [2020-04-07 20:31:38,666] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315447758946304, Destination cas:1586315320952094720, Error Count:34 [2020-04-07 20:31:38,666] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447758946304, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,666] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320952094720, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,669] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-9261, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,669] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:35 [2020-04-07 20:31:38,669] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315436512051200, Destination cas:1586315322572734464, Error Count:36 [2020-04-07 20:31:38,669] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315436512051200, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,669] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322572734464, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,677] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-28087, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,677] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:37 [2020-04-07 20:31:38,677] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315448301322240, Destination cas:1586315321544015872, Error Count:38 [2020-04-07 20:31:38,677] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315448301322240, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,677] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321544015872, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,679] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-15104, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,679] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:39 [2020-04-07 20:31:38,680] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315440859971584, Destination cas:1586315323654995968, Error Count:40 [2020-04-07 20:31:38,680] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315440859971584, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,680] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323654995968, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,681] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-23307, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,681] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:41 [2020-04-07 20:31:38,681] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315446558851072, Destination cas:1586315325065396224, Error Count:42 [2020-04-07 20:31:38,681] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315446558851072, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,681] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325065396224, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,690] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-3366, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,690] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:43 [2020-04-07 20:31:38,691] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315432048066560, Destination cas:1586315321472778240, Error Count:44 [2020-04-07 20:31:38,691] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315432048066560, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,691] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321472778240, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,694] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-24116, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,695] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:45 [2020-04-07 20:31:38,695] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315447183343616, Destination cas:1586315325241622528, Error Count:46 [2020-04-07 20:31:38,695] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447183343616, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,695] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325241622528, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,698] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-28977, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,698] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:47 [2020-04-07 20:31:38,698] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315448297586688, Destination cas:1586315321538641920, Error Count:48 [2020-04-07 20:31:38,698] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315448297586688, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,698] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321538641920, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,702] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-9455, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,702] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:49 [2020-04-07 20:31:38,702] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315436516638720, Destination cas:1586315322574700544, Error Count:50 [2020-04-07 20:31:38,702] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315436516638720, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,702] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322574700544, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,708] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16352, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,708] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:51 [2020-04-07 20:31:38,708] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315441677598720, Destination cas:1586315323823751168, Error Count:52 [2020-04-07 20:31:38,708] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441677598720, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,709] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323823751168, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,716] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-8641, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,717] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:53 [2020-04-07 20:31:38,717] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315435684921344, Destination cas:1586315322404896768, Error Count:54 [2020-04-07 20:31:38,717] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315435684921344, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,717] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322404896768, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,720] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16963, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,720] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:55 [2020-04-07 20:31:38,720] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315441698504704, Destination cas:1586315323831746560, Error Count:56 [2020-04-07 20:31:38,720] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441698504704, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,720] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323831746560, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,721] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-11083, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,721] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:57 [2020-04-07 20:31:38,722] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315438180466688, Destination cas:1586315322941833216, Error Count:58 [2020-04-07 20:31:38,722] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438180466688, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,722] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322941833216, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,725] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-484, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,725] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:59 [2020-04-07 20:31:38,725] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315429962645504, Destination cas:1586315320929026048, Error Count:60 [2020-04-07 20:31:38,725] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429962645504, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,725] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320929026048, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,726] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-9377, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,726] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:61 [2020-04-07 20:31:38,726] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315436529156096, Destination cas:1586315322578894848, Error Count:62 [2020-04-07 20:31:38,726] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315436529156096, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,726] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322578894848, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,730] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-20395, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,730] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:63 [2020-04-07 20:31:38,730] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315444584775680, Destination cas:1586315324547989504, Error Count:64 [2020-04-07 20:31:38,730] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315444584775680, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,730] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324547989504, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,731] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-25433, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,731] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:65 [2020-04-07 20:31:38,731] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315447760388096, Destination cas:1586315320956289024, Error Count:66 [2020-04-07 20:31:38,731] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447760388096, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,731] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320956289024, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,735] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16317, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,735] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:67 [2020-04-07 20:31:38,735] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315441698242560, Destination cas:1586315323831549952, Error Count:68 [2020-04-07 20:31:38,735] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441698242560, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,736] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323831549952, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,740] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-15775, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,740] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:69 [2020-04-07 20:31:38,740] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315440855121920, Destination cas:1586315323653750784, Error Count:70 [2020-04-07 20:31:38,740] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315440855121920, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,740] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323653750784, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,741] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-18248, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,741] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:71 [2020-04-07 20:31:38,741] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315443070894080, Destination cas:1586315324199010304, Error Count:72 [2020-04-07 20:31:38,741] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315443070894080, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,742] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324199010304, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,744] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-4507, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,745] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:73 [2020-04-07 20:31:38,745] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315432639725568, Destination cas:1586315321640222720, Error Count:74 [2020-04-07 20:31:38,745] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315432639725568, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,745] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321640222720, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,746] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-5363, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,746] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:75 [2020-04-07 20:31:38,746] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315433305374720, Destination cas:1586315321850527744, Error Count:76 [2020-04-07 20:31:38,746] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315433305374720, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,747] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321850527744, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,749] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-12002, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,749] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:77 [2020-04-07 20:31:38,749] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315438793949184, Destination cas:1586315323115241472, Error Count:78 [2020-04-07 20:31:38,749] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438793949184, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,749] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323115241472, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,751] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-2979, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,751] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:79 [2020-04-07 20:31:38,751] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315431350697984, Destination cas:1586315321290391552, Error Count:80 [2020-04-07 20:31:38,751] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315431350697984, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,751] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321290391552, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,756] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-27599, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,757] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:81 [2020-04-07 20:31:38,757] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315448127651840, Destination cas:1586315321360318464, Error Count:82 [2020-04-07 20:31:38,757] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315448127651840, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,757] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321360318464, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,759] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-465, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,760] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:83 [2020-04-07 20:31:38,760] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315429973590016, Destination cas:1586315320935251968, Error Count:84 [2020-04-07 20:31:38,760] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429973590016, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,760] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320935251968, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,764] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-29471, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,764] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:85 [2020-04-07 20:31:38,764] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315448476958720, Destination cas:1586315321733742592, Error Count:86 [2020-04-07 20:31:38,764] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315448476958720, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,764] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321733742592, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,766] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-6102, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,766] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:87 [2020-04-07 20:31:38,766] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315433980788736, Destination cas:1586315322015547392, Error Count:88 [2020-04-07 20:31:38,766] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315433980788736, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,766] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322015547392, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,768] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16244, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,768] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:89 [2020-04-07 20:31:38,768] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315441691623424, Destination cas:1586315323829452800, Error Count:90 [2020-04-07 20:31:38,768] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441691623424, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,768] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323829452800, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,770] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-3211, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,770] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:91 [2020-04-07 20:31:38,770] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315432033583104, Destination cas:1586315321458753536, Error Count:92 [2020-04-07 20:31:38,771] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315432033583104, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,771] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321458753536, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,782] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-11552, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,782] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:93 [2020-04-07 20:31:38,783] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315438174961664, Destination cas:1586315322938687488, Error Count:94 [2020-04-07 20:31:38,783] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438174961664, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,783] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322938687488, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,792] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-15179, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,792] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:95 [2020-04-07 20:31:38,792] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315440853876736, Destination cas:1586315323653423104, Error Count:96 [2020-04-07 20:31:38,792] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315440853876736, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,793] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323653423104, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,798] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-8253, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,798] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:97 [2020-04-07 20:31:38,799] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315435673059328, Destination cas:1586315322400047104, Error Count:98 [2020-04-07 20:31:38,799] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315435673059328, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,799] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322400047104, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,807] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-2354, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,807] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:99 [2020-04-07 20:31:38,807] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315431352008704, Destination cas:1586315321290981376, Error Count:100 [2020-04-07 20:31:38,808] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315431352008704, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,808] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321290981376, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,816] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-21135, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,817] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:101 [2020-04-07 20:31:38,817] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315445275525120, Destination cas:1586315324723232768, Error Count:102 [2020-04-07 20:31:38,817] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445275525120, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,818] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324723232768, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,819] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-19172, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,819] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:103 [2020-04-07 20:31:38,819] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315443910606848, Destination cas:1586315324371894272, Error Count:104 [2020-04-07 20:31:38,819] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315443910606848, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,819] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324371894272, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,821] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-13524, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,821] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:105 [2020-04-07 20:31:38,821] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315439421456384, Destination cas:1586315323308900352, Error Count:106 [2020-04-07 20:31:38,821] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315439421456384, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,821] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323308900352, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,826] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-23662, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,827] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:107 [2020-04-07 20:31:38,827] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315446557999104, Destination cas:1586315325064806400, Error Count:108 [2020-04-07 20:31:38,827] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315446557999104, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,827] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325064806400, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,832] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-21633, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,833] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:109 [2020-04-07 20:31:38,833] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315445277622272, Destination cas:1586315324723888128, Error Count:110 [2020-04-07 20:31:38,833] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445277622272, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,833] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324723888128, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,851] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-12723, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,851] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:111 [2020-04-07 20:31:38,852] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315438798536704, Destination cas:1586315323118714880, Error Count:112 [2020-04-07 20:31:38,852] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438798536704, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,852] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323118714880, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,856] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-22218, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,856] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:113 [2020-04-07 20:31:38,856] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315445929705472, Destination cas:1586315324890349568, Error Count:114 [2020-04-07 20:31:38,856] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445929705472, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,857] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324890349568, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,861] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16351, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,861] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:115 [2020-04-07 20:31:38,862] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315441698766848, Destination cas:1586315323832139776, Error Count:116 [2020-04-07 20:31:38,862] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441698766848, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,862] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323832139776, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,865] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-851, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,865] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:117 [2020-04-07 20:31:38,865] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315429962317824, Destination cas:1586315320928763904, Error Count:118 [2020-04-07 20:31:38,865] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429962317824, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,866] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320928763904, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,870] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-25244, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,870] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:119 [2020-04-07 20:31:38,870] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315447764582400, Destination cas:1586315320971427840, Error Count:120 [2020-04-07 20:31:38,871] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447764582400, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,871] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320971427840, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,872] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-26962, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,873] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:121 [2020-04-07 20:31:38,873] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315447945527296, Destination cas:1586315321171050496, Error Count:122 [2020-04-07 20:31:38,873] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447945527296, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,873] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321171050496, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,876] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-2481, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,876] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:123 [2020-04-07 20:31:38,876] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315431343685632, Destination cas:1586315321282461696, Error Count:124 [2020-04-07 20:31:38,877] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315431343685632, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,877] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321282461696, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,880] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-4677, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,880] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:125 [2020-04-07 20:31:38,880] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315432642871296, Destination cas:1586315321641992192, Error Count:126 [2020-04-07 20:31:38,880] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315432642871296, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,881] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321641992192, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,886] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-807, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,886] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:127 [2020-04-07 20:31:38,886] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315429965004800, Destination cas:1586315320929943552, Error Count:128 [2020-04-07 20:31:38,886] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315429965004800, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,887] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315320929943552, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,888] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-3631, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,889] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:129 [2020-04-07 20:31:38,889] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315432030175232, Destination cas:1586315321455804416, Error Count:130 [2020-04-07 20:31:38,889] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315432030175232, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,889] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321455804416, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,902] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-26108, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,902] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:131 [2020-04-07 20:31:38,902] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315447940218880, Destination cas:1586315321167249408, Error Count:132 [2020-04-07 20:31:38,903] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447940218880, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,903] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321167249408, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,904] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-24441, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,904] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:133 [2020-04-07 20:31:38,904] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315447176921088, Destination cas:1586315325235331072, Error Count:134 [2020-04-07 20:31:38,905] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447176921088, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,905] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325235331072, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,910] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-29154, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,911] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:135 [2020-04-07 20:31:38,911] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315448474271744, Destination cas:1586315321730727936, Error Count:136 [2020-04-07 20:31:38,911] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315448474271744, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,911] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321730727936, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,916] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-1234, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,916] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:137 [2020-04-07 20:31:38,917] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315430577438720, Destination cas:1586315321105711104, Error Count:138 [2020-04-07 20:31:38,917] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315430577438720, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,917] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321105711104, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,918] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-17049, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,918] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:139 [2020-04-07 20:31:38,919] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315442443649024, Destination cas:1586315323998208000, Error Count:140 [2020-04-07 20:31:38,919] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315442443649024, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,919] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323998208000, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,923] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-17482, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,923] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:141 [2020-04-07 20:31:38,923] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315442458853376, Destination cas:1586315324021145600, Error Count:142 [2020-04-07 20:31:38,923] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315442458853376, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,923] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324021145600, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,925] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-21553, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,926] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:143 [2020-04-07 20:31:38,926] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315445264449536, Destination cas:1586315324714844160, Error Count:144 [2020-04-07 20:31:38,926] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445264449536, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,926] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324714844160, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,931] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-12299, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,931] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:145 [2020-04-07 20:31:38,931] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315438791852032, Destination cas:1586315323113209856, Error Count:146 [2020-04-07 20:31:38,931] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438791852032, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,931] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323113209856, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,932] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-26166, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,932] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:147 [2020-04-07 20:31:38,932] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315447945134080, Destination cas:1586315321170788352, Error Count:148 [2020-04-07 20:31:38,933] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447945134080, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,933] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321170788352, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,936] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-15474, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,936] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:149 [2020-04-07 20:31:38,936] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315440841293824, Destination cas:1586315323648770048, Error Count:150 [2020-04-07 20:31:38,936] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315440841293824, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,937] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323648770048, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,937] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-13060, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,938] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:151 [2020-04-07 20:31:38,938] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315439420997632, Destination cas:1586315323308507136, Error Count:152 [2020-04-07 20:31:38,938] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315439420997632, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,938] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323308507136, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,941] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-12485, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,941] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:153 [2020-04-07 20:31:38,942] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315438790082560, Destination cas:1586315323111899136, Error Count:154 [2020-04-07 20:31:38,942] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438790082560, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,942] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323111899136, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,946] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-11969, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,946] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:155 [2020-04-07 20:31:38,947] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315438168276992, Destination cas:1586315322935541760, Error Count:156 [2020-04-07 20:31:38,947] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438168276992, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,947] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322935541760, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,950] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-22575, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,951] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:157 [2020-04-07 20:31:38,951] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315445920137216, Destination cas:1586315324887990272, Error Count:158 [2020-04-07 20:31:38,951] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445920137216, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,951] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324887990272, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,958] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16759, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,958] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:159 [2020-04-07 20:31:38,958] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315441690640384, Destination cas:1586315323829190656, Error Count:160 [2020-04-07 20:31:38,958] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441690640384, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,958] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323829190656, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,960] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-4233, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,960] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:161 [2020-04-07 20:31:38,960] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315432649031680, Destination cas:1586315321645072384, Error Count:162 [2020-04-07 20:31:38,960] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315432649031680, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,960] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321645072384, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,967] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-7974, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,967] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:163 [2020-04-07 20:31:38,968] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315434849402880, Destination cas:1586315322225590272, Error Count:164 [2020-04-07 20:31:38,968] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315434849402880, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,968] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322225590272, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,969] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-26868, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,969] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:165 [2020-04-07 20:31:38,969] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315447944478720, Destination cas:1586315321170264064, Error Count:166 [2020-04-07 20:31:38,969] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447944478720, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,969] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321170264064, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,970] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-17597, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,970] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:167 [2020-04-07 20:31:38,971] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315442451054592, Destination cas:1586315324008628224, Error Count:168 [2020-04-07 20:31:38,971] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315442451054592, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,971] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324008628224, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,972] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-15986, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,973] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:169 [2020-04-07 20:31:38,973] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315440853024768, Destination cas:1586315323653029888, Error Count:170 [2020-04-07 20:31:38,973] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315440853024768, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,973] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323653029888, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,976] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16344, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,976] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:171 [2020-04-07 20:31:38,976] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315441685266432, Destination cas:1586315323827290112, Error Count:172 [2020-04-07 20:31:38,976] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441685266432, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,977] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323827290112, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,979] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-29216, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,979] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:173 [2020-04-07 20:31:38,979] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315448477876224, Destination cas:1586315321734594560, Error Count:174 [2020-04-07 20:31:38,979] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315448477876224, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,979] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321734594560, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,980] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-23146, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,980] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:175 [2020-04-07 20:31:38,980] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315446561996800, Destination cas:1586315325070311424, Error Count:176 [2020-04-07 20:31:38,980] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315446561996800, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,981] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325070311424, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,983] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-19504, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,983] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:177 [2020-04-07 20:31:38,983] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315443916963840, Destination cas:1586315324376285184, Error Count:178 [2020-04-07 20:31:38,984] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315443916963840, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,984] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324376285184, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,986] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-26428, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,986] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:179 [2020-04-07 20:31:38,986] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315447941005312, Destination cas:1586315321167970304, Error Count:180 [2020-04-07 20:31:38,986] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315447941005312, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,987] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321167970304, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,989] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-16442, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,989] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:181 [2020-04-07 20:31:38,989] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315441695162368, Destination cas:1586315323830763520, Error Count:182 [2020-04-07 20:31:38,989] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315441695162368, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,989] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323830763520, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,990] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-7148, bucket:standard_bucket_1 ===== [2020-04-07 20:31:38,990] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:183 [2020-04-07 20:31:38,990] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315434855104512, Destination cas:1586315322229653504, Error Count:184 [2020-04-07 20:31:38,991] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315434855104512, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:38,991] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322229653504, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,001] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-4021, bucket:standard_bucket_1 ===== [2020-04-07 20:31:39,001] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:185 [2020-04-07 20:31:39,001] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315432649293824, Destination cas:1586315321645137920, Error Count:186 [2020-04-07 20:31:39,001] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315432649293824, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,001] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315321645137920, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,005] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-13691, bucket:standard_bucket_1 ===== [2020-04-07 20:31:39,005] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:187 [2020-04-07 20:31:39,005] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315439421063168, Destination cas:1586315323308572672, Error Count:188 [2020-04-07 20:31:39,005] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315439421063168, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,005] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323308572672, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,007] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-13515, bucket:standard_bucket_1 ===== [2020-04-07 20:31:39,007] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:189 [2020-04-07 20:31:39,007] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315439416279040, Destination cas:1586315323300380672, Error Count:190 [2020-04-07 20:31:39,008] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315439416279040, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,008] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315323300380672, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,010] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-11836, bucket:standard_bucket_1 ===== [2020-04-07 20:31:39,010] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:191 [2020-04-07 20:31:39,010] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315438180728832, Destination cas:1586315322941964288, Error Count:192 [2020-04-07 20:31:39,010] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315438180728832, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,010] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322941964288, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,014] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-23162, bucket:standard_bucket_1 ===== [2020-04-07 20:31:39,015] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:193 [2020-04-07 20:31:39,015] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315446562127872, Destination cas:1586315325070442496, Error Count:194 [2020-04-07 20:31:39,015] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315446562127872, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,015] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315325070442496, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,017] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-10168, bucket:standard_bucket_1 ===== [2020-04-07 20:31:39,017] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:195 [2020-04-07 20:31:39,017] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315437333938176, Destination cas:1586315322758135808, Error Count:196 [2020-04-07 20:31:39,018] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315437333938176, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,018] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315322758135808, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,023] - [task] [140562569275136] - ERROR - ===== Verifying rev_ids failed for key: C1-key-21635, bucket:standard_bucket_1 ===== [2020-04-07 20:31:39,024] - [task] [140562569275136] - ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:197 [2020-04-07 20:31:39,024] - [task] [140562569275136] - ERROR - cas mismatch: Source cas:1586315445277556736, Destination cas:1586315324723888128, Error Count:198 [2020-04-07 20:31:39,024] - [task] [140562569275136] - ERROR - Source meta data: {'deleted': 0, 'seqno': 2, 'cas': 1586315445277556736, 'flags': 0, 'expiration': 0} [2020-04-07 20:31:39,024] - [task] [140562569275136] - ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1586315324723888128, 'flags': 0, 'expiration': 0} [2020-04-07 20:32:13,257] - [task] [140561854215936] - INFO - 50000 items have been verified [2020-04-07 20:32:13,477] - [task] [140562569275136] - INFO - 50000 items have been verified [2020-04-07 20:32:55,071] - [task] [140562569275136] - INFO - 100000 items have been verified [2020-04-07 20:32:55,071] - [task] [140562569275136] - INFO - RevId Verification : 70000 existing items have been verified [2020-04-07 20:32:55,071] - [task] [140562569275136] - INFO - RevId Verification : 30000 deleted items have been verified [2020-04-07 20:32:55,072] - [task] [140562569275136] - INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket [2020-04-07 20:32:55,079] - [rest_client] [140562569275136] - INFO - http://172.23.98.17:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute [2020-04-07 20:32:55,480] - [task] [140561854215936] - INFO - 100000 items have been verified [2020-04-07 20:32:55,480] - [task] [140561854215936] - INFO - RevId Verification : 70000 existing items have been verified [2020-04-07 20:32:55,480] - [task] [140561854215936] - INFO - RevId Verification : 30000 deleted items have been verified [2020-04-07 20:32:55,480] - [task] [140561854215936] - INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket [2020-04-07 20:32:55,487] - [rest_client] [140561854215936] - INFO - http://172.23.98.17:8091/pools/default/buckets/default/stats?zoom=minute [2020-04-07 20:32:55,633] - [xdcrnewbasetests] [140562577667840] - INFO - Collecting logs @ 172.23.98.17 [2020-04-07 20:32:55,635] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.17 with username:root [2020-04-07 20:32:55,732] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.17 [2020-04-07 20:32:56,006] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:32:56,399] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:32:56,400] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.17: /opt/couchbase/bin/cbcollect_info 172.23.98.17-20200407-2032-diag.zip [2020-04-07 20:35:24,743] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:35:25,331] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.17: rm -f /root/172.23.98.17-20200407-2032-diag.zip [2020-04-07 20:35:25,351] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:35:25,466] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.17 with username:root [2020-04-07 20:35:25,566] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.17 [2020-04-07 20:35:25,811] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:35:26,203] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:35:26,214] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.17: tar -zcvf 172.23.98.17-472020-2035-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2020-04-07 20:35:28,314] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:35:28,722] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.17: rm -f /root/172.23.98.17-472020-2035-couch.tar.gz [2020-04-07 20:35:28,742] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:35:28,856] - [xdcrnewbasetests] [140562577667840] - INFO - Collecting logs @ 172.23.98.164 [2020-04-07 20:35:28,857] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.164 with username:root [2020-04-07 20:35:28,954] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.164 [2020-04-07 20:35:29,220] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:35:29,602] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:35:29,602] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.164: /opt/couchbase/bin/cbcollect_info 172.23.98.164-20200407-2035-diag.zip [2020-04-07 20:37:55,515] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:37:56,099] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.164: rm -f /root/172.23.98.164-20200407-2035-diag.zip [2020-04-07 20:37:56,119] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:37:56,234] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.164 with username:root [2020-04-07 20:37:56,333] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.164 [2020-04-07 20:37:56,572] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:37:56,970] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:37:56,981] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.164: tar -zcvf 172.23.98.164-472020-2037-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2020-04-07 20:37:59,331] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:37:59,804] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.164: rm -f /root/172.23.98.164-472020-2037-couch.tar.gz [2020-04-07 20:37:59,824] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:37:59,938] - [xdcrnewbasetests] [140562577667840] - INFO - Collecting logs @ 172.23.98.165 [2020-04-07 20:37:59,940] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.165 with username:root [2020-04-07 20:38:00,040] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.165 [2020-04-07 20:38:00,311] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:38:00,703] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:38:00,704] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: /opt/couchbase/bin/cbcollect_info 172.23.98.165-20200407-2037-diag.zip [2020-04-07 20:40:29,431] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:40:29,849] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: rm -f /root/172.23.98.165-20200407-2037-diag.zip [2020-04-07 20:40:29,867] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:40:29,982] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.165 with username:root [2020-04-07 20:40:30,081] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.165 [2020-04-07 20:40:30,366] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:40:30,768] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:40:30,781] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: tar -zcvf 172.23.98.165-472020-2040-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2020-04-07 20:40:33,231] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:40:33,680] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.165: rm -f /root/172.23.98.165-472020-2040-couch.tar.gz [2020-04-07 20:40:33,700] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:40:33,814] - [xdcrnewbasetests] [140562577667840] - INFO - Collecting logs @ 172.23.98.16 [2020-04-07 20:40:33,815] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.16 with username:root [2020-04-07 20:40:33,912] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.16 [2020-04-07 20:40:34,216] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:40:34,638] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:40:34,638] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.16: /opt/couchbase/bin/cbcollect_info 172.23.98.16-20200407-2040-diag.zip [2020-04-07 20:42:55,020] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:42:55,461] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.16: rm -f /root/172.23.98.16-20200407-2040-diag.zip [2020-04-07 20:42:55,481] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:42:55,596] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.16 with username:root [2020-04-07 20:42:55,695] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.16 [2020-04-07 20:42:55,966] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:42:56,365] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:42:56,376] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.16: tar -zcvf 172.23.98.16-472020-2042-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2020-04-07 20:42:58,777] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:42:59,234] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.16: rm -f /root/172.23.98.16-472020-2042-couch.tar.gz [2020-04-07 20:42:59,254] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:42:59,368] - [xdcrnewbasetests] [140562577667840] - INFO - Collecting logs @ 172.23.98.15 [2020-04-07 20:42:59,369] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.15 with username:root [2020-04-07 20:42:59,466] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.15 [2020-04-07 20:42:59,735] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:43:00,162] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:43:00,162] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.15: /opt/couchbase/bin/cbcollect_info 172.23.98.15-20200407-2042-diag.zip [2020-04-07 20:44:13,879] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:44:14,080] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.15: rm -f /root/172.23.98.15-20200407-2042-diag.zip [2020-04-07 20:44:14,098] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:44:14,213] - [remote_util] [140562577667840] - INFO - connecting to 172.23.98.15 with username:root [2020-04-07 20:44:14,312] - [remote_util] [140562577667840] - INFO - Connected to 172.23.98.15 [2020-04-07 20:44:14,596] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:44:14,997] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:44:15,008] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.15: tar -zcvf 172.23.98.15-472020-2044-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2020-04-07 20:44:15,057] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:44:15,169] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.98.15: rm -f /root/172.23.98.15-472020-2044-couch.tar.gz [2020-04-07 20:44:15,187] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:44:15,301] - [xdcrnewbasetests] [140562577667840] - INFO - Collecting logs @ 172.23.97.93 [2020-04-07 20:44:15,302] - [remote_util] [140562577667840] - INFO - connecting to 172.23.97.93 with username:root [2020-04-07 20:44:15,403] - [remote_util] [140562577667840] - INFO - Connected to 172.23.97.93 [2020-04-07 20:44:15,649] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:44:15,993] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:44:15,994] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.97.93: /opt/couchbase/bin/cbcollect_info 172.23.97.93-20200407-2044-diag.zip [2020-04-07 20:45:54,860] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:45:55,073] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.97.93: rm -f /root/172.23.97.93-20200407-2044-diag.zip [2020-04-07 20:45:55,091] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:45:55,206] - [remote_util] [140562577667840] - INFO - connecting to 172.23.97.93 with username:root [2020-04-07 20:45:55,306] - [remote_util] [140562577667840] - INFO - Connected to 172.23.97.93 [2020-04-07 20:45:55,591] - [remote_util] [140562577667840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-04-07 20:45:55,990] - [remote_util] [140562577667840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-04-07 20:45:56,000] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.97.93: tar -zcvf 172.23.97.93-472020-2045-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2020-04-07 20:45:56,048] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:45:56,157] - [remote_util] [140562577667840] - INFO - running command.raw on 172.23.97.93: rm -f /root/172.23.97.93-472020-2045-couch.tar.gz [2020-04-07 20:45:56,175] - [remote_util] [140562577667840] - INFO - command executed successfully [2020-04-07 20:45:56,329] - [xdcrnewbasetests] [140562577667840] - INFO - ==== XDCRNewbasetests cleanup is started for test #10 load_with_async_ops_with_warmup_master ==== [2020-04-07 20:45:56,329] - [xdcrnewbasetests] [140562577667840] - INFO - removing xdcr/nodes settings [2020-04-07 20:45:56,344] - [rest_client] [140562577667840] - INFO - Deleting replication /controller/cancelXDCR/32a5a3fe83d9f903bcf55ade47abaa6f%2Fdefault%2Fdefault [2020-04-07 20:45:56,351] - [rest_client] [140562577667840] - INFO - Replication deleted successfully [2020-04-07 20:45:56,351] - [rest_client] [140562577667840] - INFO - Deleting replication /controller/cancelXDCR/32a5a3fe83d9f903bcf55ade47abaa6f%2Fstandard_bucket_1%2Fstandard_bucket_1 [2020-04-07 20:45:56,367] - [rest_client] [140562577667840] - INFO - Replication deleted successfully [2020-04-07 20:45:56,374] - [rest_client] [140562577667840] - INFO - removing remote cluster name:remote_cluster_C1-C2 [2020-04-07 20:45:56,393] - [xdcrnewbasetests] [140562577667840] - INFO - cleanup [ip:172.23.98.17 port:8091 ssh_username:root, ip:172.23.98.164 port:8091 ssh_username:root] [2020-04-07 20:45:56,422] - [bucket_helper] [140562577667840] - INFO - deleting existing buckets [u'default', u'standard_bucket_1'] on 172.23.98.17 [2020-04-07 20:45:56,422] - [bucket_helper] [140562577667840] - INFO - remove bucket default ... [2020-04-07 20:45:58,073] - [bucket_helper] [140562577667840] - INFO - deleted bucket : default from 172.23.98.17 [2020-04-07 20:45:58,073] - [bucket_helper] [140562577667840] - INFO - waiting for bucket deletion to complete.... [2020-04-07 20:45:58,089] - [rest_client] [140562577667840] - INFO - node 172.23.98.17 existing buckets : [u'standard_bucket_1'] [2020-04-07 20:45:58,090] - [bucket_helper] [140562577667840] - INFO - remove bucket standard_bucket_1 ... [2020-04-07 20:45:59,554] - [bucket_helper] [140562577667840] - INFO - deleted bucket : standard_bucket_1 from 172.23.98.17 [2020-04-07 20:45:59,554] - [bucket_helper] [140562577667840] - INFO - waiting for bucket deletion to complete.... [2020-04-07 20:45:59,558] - [rest_client] [140562577667840] - INFO - node 172.23.98.17 existing buckets : [] [2020-04-07 20:45:59,558] - [bucket_helper] [140562577667840] - INFO - sleep 2 seconds to make sure all buckets ([u'default', u'standard_bucket_1']) were deleted completely. [2020-04-07 20:46:01,581] - [cluster_helper] [140562577667840] - INFO - rebalancing all nodes in order to remove nodes [2020-04-07 20:46:01,585] - [rest_client] [140562577667840] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.98.164', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.98.17,ns_1@172.23.98.164'} [2020-04-07 20:46:01,590] - [rest_client] [140562577667840] - INFO - rebalance operation started [2020-04-07 20:46:01,596] - [rest_client] [140562577667840] - INFO - rebalance percentage : 0.00 % [2020-04-07 20:46:21,619] - [rest_client] [140562577667840] - INFO - rebalance progress took 20.03 seconds [2020-04-07 20:46:21,619] - [rest_client] [140562577667840] - INFO - sleep for 10 seconds after rebalance... [2020-04-07 20:46:31,639] - [cluster_helper] [140562577667840] - INFO - removed all the nodes from cluster associated with ip:172.23.98.17 port:8091 ssh_username:root ? [(u'ns_1@172.23.98.164', 8091)] [2020-04-07 20:46:31,644] - [cluster_helper] [140562577667840] - INFO - waiting for ns_server @ 172.23.98.17:8091 [2020-04-07 20:46:31,649] - [cluster_helper] [140562577667840] - INFO - ns_server @ 172.23.98.17:8091 is running [2020-04-07 20:46:31,677] - [cluster_helper] [140562577667840] - INFO - waiting for ns_server @ 172.23.98.164:8091 [2020-04-07 20:46:31,681] - [cluster_helper] [140562577667840] - INFO - ns_server @ 172.23.98.164:8091 is running [2020-04-07 20:46:31,681] - [xdcrnewbasetests] [140562577667840] - INFO - removing xdcr/nodes settings [2020-04-07 20:46:31,713] - [xdcrnewbasetests] [140562577667840] - INFO - cleanup [ip:172.23.98.165 port:8091 ssh_username:root, ip:172.23.98.16 port:8091 ssh_username:root] [2020-04-07 20:46:31,742] - [bucket_helper] [140562577667840] - INFO - deleting existing buckets [u'default', u'standard_bucket_1'] on 172.23.98.165 [2020-04-07 20:46:31,742] - [bucket_helper] [140562577667840] - INFO - remove bucket default ... [2020-04-07 20:46:33,069] - [bucket_helper] [140562577667840] - INFO - deleted bucket : default from 172.23.98.165 [2020-04-07 20:46:33,069] - [bucket_helper] [140562577667840] - INFO - waiting for bucket deletion to complete.... [2020-04-07 20:46:33,087] - [rest_client] [140562577667840] - INFO - node 172.23.98.165 existing buckets : [u'standard_bucket_1'] [2020-04-07 20:46:33,088] - [bucket_helper] [140562577667840] - INFO - remove bucket standard_bucket_1 ... [2020-04-07 20:46:33,822] - [bucket_helper] [140562577667840] - INFO - deleted bucket : standard_bucket_1 from 172.23.98.165 [2020-04-07 20:46:33,822] - [bucket_helper] [140562577667840] - INFO - waiting for bucket deletion to complete.... [2020-04-07 20:46:33,826] - [rest_client] [140562577667840] - INFO - node 172.23.98.165 existing buckets : [] [2020-04-07 20:46:33,827] - [bucket_helper] [140562577667840] - INFO - sleep 2 seconds to make sure all buckets ([u'default', u'standard_bucket_1']) were deleted completely. [2020-04-07 20:46:35,850] - [cluster_helper] [140562577667840] - INFO - rebalancing all nodes in order to remove nodes [2020-04-07 20:46:35,854] - [rest_client] [140562577667840] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.98.16', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.98.16,ns_1@172.23.98.165'} [2020-04-07 20:46:35,859] - [rest_client] [140562577667840] - INFO - rebalance operation started [2020-04-07 20:46:35,866] - [rest_client] [140562577667840] - INFO - rebalance percentage : 0.00 % [2020-04-07 20:46:55,895] - [rest_client] [140562577667840] - INFO - rebalance progress took 20.04 seconds [2020-04-07 20:46:55,895] - [rest_client] [140562577667840] - INFO - sleep for 10 seconds after rebalance... [2020-04-07 20:47:05,915] - [cluster_helper] [140562577667840] - INFO - removed all the nodes from cluster associated with ip:172.23.98.165 port:8091 ssh_username:root ? [(u'ns_1@172.23.98.16', 8091)] [2020-04-07 20:47:05,921] - [cluster_helper] [140562577667840] - INFO - waiting for ns_server @ 172.23.98.165:8091 [2020-04-07 20:47:05,926] - [cluster_helper] [140562577667840] - INFO - ns_server @ 172.23.98.165:8091 is running [2020-04-07 20:47:05,952] - [cluster_helper] [140562577667840] - INFO - waiting for ns_server @ 172.23.98.16:8091 [2020-04-07 20:47:05,956] - [cluster_helper] [140562577667840] - INFO - ns_server @ 172.23.98.16:8091 is running [2020-04-07 20:47:05,957] - [xdcrnewbasetests] [140562577667840] - INFO - ==== XDCRNewbasetests cleanup is finished for test #10 load_with_async_ops_with_warmup_master ===