Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8346

memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 255

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 2.1.0
    • 2.1.0
    • couchbase-bucket
    • Security Level: Public
    • None
    • windows 2008 R2 64bit
    • Windows 64-bit

    Description

      Run test with build 2.0.2-807

      ./testrunner -i /tmp/6-win-xdcr.ini get-cbcollect-info=True -t xdcr.biXDCR.bidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=bidirection,doc-ops=create-update-delete,doc-ops-dest=create,update,failover=source

      Test failed and I saw a lot memcached crashed in log page

      Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 255. Restarting. Messages: Mon May 20 17:02:36.236179 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 83 was completed succecssfully.
      Mon May 20 17:02:36.236179 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 84 was completed succecssfully.
      Mon May 20 17:02:36.241062 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 85 was completed succecssfully.
      Mon May 20 17:02:36.242039 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 86 was completed succecssfully.
      Mon May 20 17:02:36.243992 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 87 was completed succecssfully.
      Mon May 20 17:02:36.243992 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 88 was completed succecssfully.
      Mon May 20 17:02:36.246921 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 89 was completed succecssfully.
      Mon May 20 17:02:36.259617 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 90 was completed succecssfully.
      Mon May 20 17:02:36.259617 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 91 was completed succecssfully.
      Mon May 20 17:02:36.259617 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 92 was completed succecssfully.
      Mon May 20 17:02:36.265476 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 93 was completed succecssfully.
      Mon May 20 17:02:36.265476 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 94 was completed succecssfully.
      Mon May 20 17:02:36.265476 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 95 was completed succecssfully.
      Mon May 20 17:02:36.266453 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 96 was completed succecssfully.
      Mon May 20 17:02:36.266453 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 97 was completed succecssfully.
      Mon May 20 17:02:36.268406 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 98 was completed succecssfully.
      Mon May 20 17:02:36.275242 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 99 was completed succecssfully.
      Mon May 20 17:02:36.275242 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 100 was completed succecssfully.
      Mon May 20 17:02:36.276218 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 101 was completed succecssfully.
      Mon May 20 17:02:36.276218 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 102 was completed succecssfully.
      Mon May 20 17:02:36.276218 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 103 was completed succecssfully.
      Mon May 20 17:02:36.277195 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 104 was completed succecssfully.
      Mon May 20 17:02:36.277195 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 105 was completed succecssfully.
      Mon May 20 17:02:36.278171 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 106 was completed succecssfully.
      Mon May 20 17:02:36.278171 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 107 was completed succecssfully.
      Mon May 20 17:02:36.279148 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 108 was completed succecssfully.
      Mon May 20 17:02:36.283054 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 109 was completed succecssfully.
      Mon May 20 17:02:36.285007 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 110 was completed succecssfully.
      Mon May 20 17:02:36.286960 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 111 was completed succecssfully.
      Mon May 20 17:02:36.286960 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 112 was completed succecssfully.
      Mon May 20 17:02:36.296726 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 113 was completed succecssfully.
      Mon May 20 17:02:36.296726 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 114 was completed succecssfully.
      Mon May 20 17:02:36.301609 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 115 was completed succecssfully.
      Mon May 20 17:02:36.317234 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 116 was completed succecssfully.
      Mon May 20 17:02:36.317234 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 117 was completed succecssfully.
      Mon May 20 17:02:36.326023 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 118 was completed succecssfully.
      Mon May 20 17:02:36.326023 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 119 was completed succecssfully.
      Mon May 20 17:02:36.326023 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 120 was completed succecssfully.
      Mon May 20 17:02:36.326023 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 121 was completed succecssfully.
      Mon May 20 17:02:36.326023 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 122 was completed succecssfully.
      Mon May 20 17:02:36.327000 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 123 was completed succecssfully.
      Mon May 20 17:02:36.327000 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 124 was completed succecssfully.
      Mon May 20 17:02:36.328953 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 125 was completed succecssfully.
      Mon May 20 17:02:36.328953 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 126 was completed succecssfully.
      Mon May 20 17:02:36.329929 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 127 was completed succecssfully.
      Mon May 20 17:02:36.329929 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 128 was completed succecssfully.
      Mon May 20 17:02:36.329929 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 129 was completed succecssfully.
      Mon May 20 17:02:36.332859 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 130 was completed succecssfully.
      Mon May 20 17:02:36.333835 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 131 was completed succecssfully.
      Mon May 20 17:02:36.333835 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 132 was completed succecssfully.
      Mon May 20 17:02:36.333835 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 133 was completed succecssfully.
      Mon May 20 17:02:36.333835 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 134 was completed succecssfully.
      Mon May 20 17:02:36.334812 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 135 was completed succecssfully.
      Mon May 20 17:02:36.340671 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 136 was completed succecssfully.
      Mon May 20 17:02:36.340671 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 137 was completed succecssfully.
      Mon May 20 17:02:36.343601 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 138 was completed succecssfully.
      Mon May 20 17:02:36.362156 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 139 was completed succecssfully.
      Mon May 20 17:02:36.366062 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 140 was completed succecssfully.
      Mon May 20 17:02:36.366062 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 141 was completed succecssfully.
      Mon May 20 17:02:36.366062 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 142 was completed succecssfully.
      Mon May 20 17:02:36.369968 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 143 was completed succecssfully.
      Mon May 20 17:02:36.369968 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 144 was completed succecssfully.
      Mon May 20 17:02:36.369968 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 145 was completed succecssfully.
      Mon May 20 17:02:36.371921 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 146 was completed succecssfully.
      Mon May 20 17:02:36.371921 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 147 was completed succecssfully.
      Mon May 20 17:02:36.372898 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 148 was completed succecssfully.
      Mon May 20 17:02:36.373875 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 149 was completed succecssfully.
      Mon May 20 17:02:36.373875 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 150 was completed succecssfully.
      Mon May 20 17:02:36.392429 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 151 was completed succecssfully.
      Mon May 20 17:02:36.396335 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 152 was completed succecssfully.
      Mon May 20 17:02:36.396335 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 153 was completed succecssfully.
      Mon May 20 17:02:36.396335 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 154 was completed succecssfully.
      Mon May 20 17:02:36.396335 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 155 was completed succecssfully.
      Mon May 20 17:02:36.396335 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 156 was completed succecssfully.
      Mon May 20 17:02:36.396335 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 157 was completed succecssfully.
      Mon May 20 17:02:36.398289 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 158 was completed succecssfully.
      Mon May 20 17:02:36.398289 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 159 was completed succecssfully.
      Mon May 20 17:02:36.399265 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 160 was completed succecssfully.
      Mon May 20 17:02:36.399265 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 161 was completed succecssfully.
      Mon May 20 17:02:36.403171 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 162 was completed succecssfully.
      Mon May 20 17:02:36.403171 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 163 was completed succecssfully.
      Mon May 20 17:02:36.406101 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 164 was completed succecssfully.
      Mon May 20 17:02:36.407078 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 165 was completed succecssfully.
      Mon May 20 17:02:36.407078 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 166 was completed succecssfully.
      Mon May 20 17:02:36.409031 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 167 was completed succecssfully.
      Mon May 20 17:02:36.412937 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 168 was completed succecssfully.
      Mon May 20 17:02:36.412937 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 169 was completed succecssfully.
      Mon May 20 17:02:36.425632 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 170 was completed succecssfully.
      Mon May 20 17:05:23.420750 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_29 - disconnected
      Mon May 20 17:05:23.825046 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:anon_29"
      Mon May 20 17:05:23.987156 Pacific Daylight Time 3: (default) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.125 - disconnected, keep alive for 300 seconds
      EOL on stdin. Initiating shutdown
      Mon May 20 17:05:53.190281 Pacific Daylight Time 3: (default) Shutting down tap connections!
      Mon May 20 17:05:53.191257 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.125"
      Mon May 20 17:05:53.205906 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.123"
      Mon May 20 17:05:53.205906 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:anon_30"
      Mon May 20 17:05:54.208835 Pacific Daylight Time 3: (default) Had to wait 999 ms for shutdown
      Mon May 20 17:05:55.209812 Pacific Daylight Time 3: (default) Had to wait 1000 ms for shutdown
      Mon May 20 17:05:56.210789 Pacific Daylight Time 3: (default) Had to wait 1000 ms for shutdown
      Mon May 20 17:05:57.212742 Pacific Daylight Time 3: (default) Had to wait 1001 ms for shutdown ns_log000 ns_1@10.3.2.124 17:06:25 - Mon May 20, 2013
      Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.3.2.124 17:06:09 - Mon May 20, 2013
      Node 'ns_1@10.3.2.124' saw that node 'ns_1@10.3.2.123' went down. Details: [

      {nodedown_reason, connection_closed}

      ] ns_node_disco005 ns_1@10.3.2.124 17:06:01 - Mon May 20, 2013

      Link to manifest file of this build
      http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-807-rel.setup.exe.manifest.xml

      This bug may relate to bug MB-8296

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              thuan Thuan Nguyen
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty