Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-28626

memcached exits with status 134 and rebalance failures in centos longevity

    XMLWordPrintable

Details

    Description

      Centos longevity was kicked off with 5.5.0-2126 which had fixes for MB-28453 - observing memcached exits with status code 71 and 134 now - all rebalances are failing - opening a new bug as the other bug is being closed with unit tests and there is a new status code for the exit now:

      2018-03-11T08:29:37.553-07:00, ns_log:0:info:message(ns_1@172.23.108.104) - Service 'memcached' exited with status 71. Restarting. Messages:
      		 
      2018-03-11T08:29:37.154074Z INFO Setting SSL minimum protocol to: tlsv1
      		 
      2018-03-11T08:29:37.155162Z INFO Breakpad enabled. Minidumps will be written to '/opt/couchbase/var/lib/couchbase/crash'
      		 
      2018-03-11T08:29:37.155373Z INFO Setting SSL minimum protocol to: tlsv1
      		 
      2018-03-11T08:29:37.161012Z INFO Disabling Breakpad
      		 
      2018-03-11T08:29:37.161118Z INFO Breakpad enabled. Minidumps will be written to '/opt/couchbase/var/lib/couchbase/crash'
      		 
      2018-03-11T08:29:37.520208Z CRITICAL Failed to create listening socket
      		 
      2018-03-11T08:29:40.007-07:00, ns_memcached:0:info:message(ns_1@172.23.98.135) - Control connection to memcached on 'ns_1@172.23.98.135' disconnected: {error,
      		 
                                                                             closed}
      		 
      2018-03-11T08:29:40.009-07:00, ns_log:0:info:message(ns_1@172.23.98.135) - Service 'memcached' exited with status 134. Restarting. Messages:
      		 
      2018-03-11T08:29:39.613980Z CRITICAL     /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so(couchstore_changes_since+0x1d8) [0x7ff5d4b93000+0x13658]
      		 
      2018-03-11T08:29:39.613986Z CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7ff5d4dd4000+0x15fd2b]
      		 
      2018-03-11T08:29:39.613991Z CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7ff5d4dd4000+0x5036a]
      		 
      2018-03-11T08:29:39.613996Z CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7ff5d4dd4000+0x51678]
      		 
      2018-03-11T08:29:39.614002Z CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7ff5d4dd4000+0x4d6b2]
      		 
      2018-03-11T08:29:39.614006Z CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7ff5d4dd4000+0x4d9fd]
      		 
      2018-03-11T08:29:39.614011Z CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7ff5d4dd4000+0xc0699]
      		 
      2018-03-11T08:29:39.614016Z CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7ff5eb338000+0x88e7]
      		 
      2018-03-11T08:29:39.614022Z CRITICAL     /lib64/libpthread.so.0() [0x7ff5eacf8000+0x7dc5]
      		 
      2018-03-11T08:29:39.614051Z CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7ff5e8d2e000+0xf621d]

      Attachments

        1. 01e93fb4-e197-3fb6-151614fe-421c64c5.dmp
          440 kB
          Arunkumar Senthilnathan
        2. 23da6161-6a5b-b438-4a371e95-7d583b64.dmp
          356 kB
          Arunkumar Senthilnathan
        3. 367fc851-a0b7-baa0-598a678a-7c048026.dmp
          366 kB
          Arunkumar Senthilnathan
        4. 4a90b673-3c3d-810c-028c9a04-6132a55c.dmp
          334 kB
          Arunkumar Senthilnathan
        5. 6f83a17c-2778-2c16-565b679a-7ff1bd48.dmp
          390 kB
          Arunkumar Senthilnathan
        6. Screen Shot 2018-03-11 at 1.34.42 PM.png
          141 kB
          Arunkumar Senthilnathan
        7. Screen Shot 2018-03-11 at 1.34.58 PM.png
          250 kB
          Arunkumar Senthilnathan

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-28630 exception occurred in runloop - closing connection: bad_function_call

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-28453 memcached exits with status 134 and rebalance failures in centos longevity

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-28487 __cxa_pure_virtual called in run_event_loop()

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-28626
          # Subject Branch Project Status CR V
          90868,4 MB-28626: Log checkpointList size when exception occurs master kv_engine Status: MERGED +2 +1
          90917,6 MB-28626: Add debug code to check the snap start/end ranges master kv_engine Status: MERGED +2 +1
          90936,4 Revert "MB-25822: Do not lock on 'releaseLock' when the ConnNofierCallback runs" master kv_engine Status: MERGED +2 +1
          91740,2 MB-28707: Revert "MB-28626: Add debug code to snapshot ranges" master kv_engine Status: MERGED +2 +1

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  PagerDuty