Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Blocker
Fix Version/s: 4.0.0
Affects Version/s: 4.0.0
Component/s: XDCR
Security Level: Public
Labels:
- ga
- system-test

Triage:
Untriaged
Operating System:
Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump:
https://s3.amazonaws.com/cb-customers/Aruna/collectinfo-2015-07-23T164530-ns_1%40172.23.105.52.zip
Epic Link:
XDCR next release
Is this a Regression?:
Unknown

Description

Build

4.0.0-3490

Found in kv+xdcr+tpcc system test

Attaching screenshots from ITEM bucket on http://172.23.105.44:8091/ on which there is no replication/dcp activity.
from outbound mutations per server and percent complete(by server) it appears .52 has not sent any mutations over

StatisticsManager 2015-07-23T05:49:51.941-07:00 [INFO] 424a76c2626a41770face72073ec50eb/ITEM/ITEM total_docs=4026888, docs_processed=0, changes_left=4026888

StatisticsManager 2015-07-23T05:49:52.940-07:00 [INFO] 424a76c2626a41770face72073ec50eb/ITEM/ITEM total_docs=4026888, docs_processed=0, changes_left=4026888

StatisticsManager 2015-07-23T05:49:53.939-07:00 [INFO] 424a76c2626a41770face72073ec50eb/ITEM/ITEM total_docs=4026888, docs_processed=0, changes_left=4026888

StatisticsManager 2015-07-23T09:49:11.943-07:00 [INFO] 424a76c2626a41770face72073ec50eb/ITEM/ITEM total_docs=4026888, docs_processed=0, changes_left=4026888

StatisticsManager 2015-07-23T09:49:12.943-07:00 [INFO] 424a76c2626a41770face72073ec50eb/ITEM/ITEM total_docs=4026888, docs_processed=0, changes_left=4026888

From the last stats manager log, I see same errors as ~~MB-15805~~. But the last error was recorded at 2015-07-22T20:12:39 (13 hrs earlier). Now the pipeline is in "replicating" state, yet there is no replication activity or errors reported, why so?

StatisticsManager 2015-07-23T09:49:10.940-07:00 [INFO] 424a76c2626a41770face72073ec50eb/ITEM/ITEM total_docs=4026888, docs_processed=0, changes_left=4026888

 424a76c2626a41770face72073ec50eb/ITEM/ITEM:name={424a76c2626a41770face72073ec50eb/ITEM/ITEM}, status={Replicating}, errors={[{"time":"2015-07-22T20:12:39.660635534-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:12:24.227799012-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:12:08.751523265-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:11:37.343677328-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:11:17.115208646-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:11:01.542528094-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Received error response from memcached in target cluster."},{"time":"2015-07-22T20:10:46.619748106-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:10:23.649474291-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Received error response from memcached in target cluster."},{"time":"2015-07-22T20:10:09.20625786-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:09:47.501890628-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:09:29.724097897-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:09:14.036246692-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:08:56.883375208-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:08:34.642848288-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Received error response from memcached in target cluster."},{"time":"2015-07-22T20:08:18.35975066-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:07:53.072761302-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:07:36.728227171-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:07:09.945321359-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:06:53.296982857-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"},{"time":"2015-07-22T20:06:37.933854408-07:00","errMsg":"xmem_424a76c2626a41770face72073ec50eb/ITEM/ITEM_172.23.105.61:11210_0:Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}"}]}, progress={Pipeline is running}