Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-15177

Node failing to come up, showing "metadata corruption"

    XMLWordPrintable

Details

    Description

      I'm having a hard time understanding why this node won't start up, but I see errors in the log along the lines of:
      [ns_server:debug,2015-05-28T18:05:00.604Z,babysitter_of_ns_1@127.0.0.1:ns_crash_log<0.64.0>:ns_crash_log:handle_cast:63]Dropping oldest unconsumed crash:

      {goxdcr,'babysitter_of_ns_1@127.0.0.1',0, "ReplicationSpecChangeListener 2015-05-28T17:56:33.635Z [INFO] metakv.RunObserveChildren failed, err=Post http://127.0.0.1:8091/_metakv: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connection refused\nMetadataService 2015-05-28T17:56:33.635Z [ERROR] metakv.ListAllChildren failed. path=/replicationSpec/, err=Post http://127.0.0.1:8091/_metakv: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connection refused, num_of_retry=2\nReplicationManager 2015-05-28T17:56:33.635Z [INFO] Replication manager is exiting...\nRemoteClusterChangeListener 2015-05-28T17:56:33.635Z [INFO] metakv.RunObserveChildren failed, err=Post http://127.0.0.1:8091/_metakv: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connection refused\nReplicationManager 2015-05-28T17:56:33.635Z [INFO] Replication manager is already in the processof stopping, no-op on this stop request"}

      [error_logger:error,2015-05-28T18:05:00.604Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.8958.0> terminating

        • Last message in was {#Port<0.9873>,
          Unknown macro: {exit_status,0}

          }

        • When Server state ==
          Unknown macro: {state,#Port<0.9873>,goxdcr, {["ReplicationManager 2015-05-28T18:05:00.603Z [INFO] Replication manager is already in the processof stopping, no-op on this stop request", "ReplicationSpecService 2015-05-28T18:05:00.603Z [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5", "ReplicationManager 2015-05-28T18:05:00.603Z [INFO] Replication manager is already in the processof stopping, no-op on this stop request", "MetadataService 2015-05-28T18:05:00.603Z [ERROR] metakv.ListAllChildren failed. path=/replicationSpec/, err=Post http://127.0.0.1:8091/_metakv: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connection refused, num_of_retry=4"], ["ReplicationManager 2015-05-28T18:05:00.603Z [INFO] Replication manager is exiting..."]}, goxdcr,undefined,[],0}
        • Reason for termination ==
        • {abnormal,0}

      Log is at:
      http://customers.couchbase.com.s3.amazonaws.com/perry/crash.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            parag Parag Agarwal (Inactive)
            perry Perry Krug
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty