Details
-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
2.1.0
-
Security Level: Public
-
None
Description
Encountered this issue during an xdcr test on 2.0.2 build 780.
Created 3 buckets on local/remote site.
Started unidirection replication from default bucket on local site to remote.
I waited about 5 minutes after xdcr pairing to start loading data. Then I checked on the test about 30 minutes into the access phase I notices nodes were constantly restarting with these messages in UI logs:
Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 134. Restarting.
and from babysitter.1:
[ns_server:info,2013-05-06T16:36:18.200,babysitter_of_ns_1@127.0.0.1:<0.134.0>:ns_port_server:log:168]memcached<0.134.0>: memcached: src/c
ouch-kvstore/couch-kvstore.cc:1331: static int CouchKVStore::recordDbDump(Db*, DocInfo*, void*): Assertion `metadata.size == 16' failed.
[ns_server:info,2013-05-06T16:36:18.307,babysitter_of_ns_1@127.0.0.1:<0.133.0>:supervisor_cushion:handle_info:58]Cushion managed superviso
r for memcached failed:
[ns_server:debug,2013-05-06T16:36:18.308,babysitter_of_ns_1@127.0.0.1:<0.135.0>:supervisor_cushion:init:39]starting ns_port_server with de
lay of 5000
[error_logger:error,2013-05-06T16:36:18.307,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Generi
c server <0.134.0> terminating
-
- Last message in was {#Port<0.3039>,
Unknown macro: {exit_status,134}
}
- When Server state == {state,#Port<0.3039>,memcached,
{["memcached: src/couch-kvstore/couch-kvstore.cc:1331: static int CouchKVStore::recordDbDump(Db*, DocInfo*,
void*): Assertion `metadata.size == 16' failed.",
"Mon May 6 16:36:15.660505 PDT 3: (saslbucket) metadata loaded in 2819 ms",
"Mon May 6 16:36:13.707220 PDT 3: (default) warmup completed in 804 ms",
"Mon May 6 16:36:13.694935 PDT 3: (default) metadata loaded in 792 ms",
"Mon May 6 16:36:13.018266 PDT 3: (default) Failed to load mutation log, falling back to key dump",
"Mon May 6 16:36:12.942281 PDT 3: Extension support isn't implemented in this version of bucket_engine",
"Mon May 6 16:36:12.936368 PDT 3: (saslbucket) Failed to load mutation log, falling back to key dump",
"Mon May 6 16:36:12.886616 PDT 3: (default) Connected to mccouch: \"127.0.0.1:11213\"",
"Mon May 6 16:36:12.885946 PDT 3: (default) Trying to connect to mccouch: \"127.0.0.1:11213\"",
"Mon May 6 16:36:12.879580 PDT 3: Extension support isn't implemented in this version of bucket_engine",
"Mon May 6 16:36:12.844079 PDT 3: (saslbucket1) Failed to load mutation log, falling back to key dump",
"Mon May 6 16:36:12.834687 PDT 3: (saslbucket) Connected to mccouch: \"127.0.0.1:11213\"",
"Mon May 6 16:36:12.834366 PDT 3: (saslbucket) Trying to connect to mccouch: \"127.0.0.1:11213\"",
"Mon May 6 16:36:12.810127 PDT 3: Extension support isn't implemented in this version of bucket_engine",
"Mon May 6 16:36:12.767939 PDT 3: (saslbucket1) Connected to mccouch: \"127.0.0.1:11213\"",
"Mon May 6 16:36:12.767549 PDT 3: (saslbucket1) Trying to connect to mccouch: \"127.0.0.1:11213\"",
empty],
- Last message in was {#Port<0.3039>,
Seems the cluster never stabilizes but memcached keeps restarting.
Logs from suspected host attached(172.23.105.45 ), remaining logs pending.