Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7652

2.0.2 memcached crashes in EventuallyPersistentStore

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.0
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None

      Description

      This bug has been formed as a separate issue with crashes in EventuallyPersistentStore for 2.0.2 builds from MB-7509

      (Removed 2.0.1 stack trace since this bug is about 2.0.2)

      (Removed 2.0.1 stack trace since this bug is about 2.0.2)

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        andreibaranouski Andrei Baranouski added a comment -

        Mike, I got the same behaviour and have a question:

        yes, indeed on one node was out of space on disk when I checked

        [jenkins@cen-1709 ~]$ df
        Filesystem 1K-blocks Used Available Use% Mounted on
        /dev/mapper/VolGroup00-LogVol00
        55007284 51424616 743324 99% /
        /dev/sda1 101086 19983 75884 21% /boot
        tmpfs 2021844 0 2021844 0% /dev/shm

        but this was achieved when the memcached started to generate a lot of core dumps:

        10.5.2.13
        [jenkins@cen-1709 tmp]$ ls -la
        total 41887784
        drwxrwxrwt 5 couchbase couchbase 20480 Mar 5 02:17 .
        drwxr-xr-x 25 root root 4096 Feb 18 03:09 ..
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:07 core.memcached.10010
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:07 core.memcached.10369
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:07 core.memcached.10426
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:07 core.memcached.10440
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:07 core.memcached.10454
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:08 core.memcached.10470
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:09 core.memcached.10485
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:09 core.memcached.10502
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:09 core.memcached.10516
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:09 core.memcached.10530
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:09 core.memcached.10544
        rw------ 1 couchbase couchbase 250097664 Mar 4 13:09 core.memcached.10558
        rw------ 1 couchbase couchbase 250093568 Mar 4 13:10 core.memcached.10572
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:10 core.memcached.10587
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:10 core.memcached.10603
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:10 core.memcached.10617
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:10 core.memcached.10631
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:10 core.memcached.10645
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:11 core.memcached.10659
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:11 core.memcached.10674
        rw------ 1 couchbase couchbase 250114048 Mar 4 13:11 core.memcached.10688
        rw------ 1 couchbase couchbase 250109952 Mar 4 13:11 core.memcached.10702
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:11 core.memcached.10718
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:11 core.memcached.10732
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:12 core.memcached.10746
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:22 core.memcached.1076
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:12 core.memcached.10761
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:12 core.memcached.10775
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:12 core.memcached.10789
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:12 core.memcached.10803
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:12 core.memcached.10819
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:13 core.memcached.10833
        rw------ 1 couchbase couchbase 250114048 Mar 4 13:13 core.memcached.10848
        rw------ 1 couchbase couchbase 250097664 Mar 4 13:13 core.memcached.11068
        rw------ 1 couchbase couchbase 250097664 Mar 4 13:13 core.memcached.11293
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:13 core.memcached.11546
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:13 core.memcached.11911
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:14 core.memcached.11942
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:14 core.memcached.11959
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:14 core.memcached.11973
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:14 core.memcached.11987
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:14 core.memcached.12001
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:14 core.memcached.12016
        rw------ 1 couchbase couchbase 250093568 Mar 4 13:15 core.memcached.12030
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:15 core.memcached.12045
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:15 core.memcached.12061
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:15 core.memcached.12075
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:15 core.memcached.12089
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:15 core.memcached.12103
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:16 core.memcached.12117
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:16 core.memcached.12132
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:16 core.memcached.12146
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:16 core.memcached.12160
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:16 core.memcached.12176
        rw------ 1 couchbase couchbase 250097664 Mar 4 13:16 core.memcached.12190
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:17 core.memcached.12204
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:17 core.memcached.12219
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:17 core.memcached.12233
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:17 core.memcached.12247
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:17 core.memcached.12261
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:17 core.memcached.12277
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:18 core.memcached.12291
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:18 core.memcached.12306
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:18 core.memcached.12320
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:18 core.memcached.12502
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:18 core.memcached.12516
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:18 core.memcached.12530
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:19 core.memcached.12544
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:19 core.memcached.12561
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:19 core.memcached.12575
        rw------ 1 couchbase couchbase 250097664 Mar 4 13:19 core.memcached.12843
        rw------ 1 couchbase couchbase 250093568 Mar 4 13:19 core.memcached.13109
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:20 core.memcached.13300
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:21 core.memcached.13490
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:21 core.memcached.13505
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:21 core.memcached.13519
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:21 core.memcached.13533
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:21 core.memcached.13549
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:21 core.memcached.13563
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:22 core.memcached.13577
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:22 core.memcached.13592
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:22 core.memcached.13606
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:22 core.memcached.13620
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:22 core.memcached.13634
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:22 core.memcached.13650
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:23 core.memcached.13664
        rw------ 1 couchbase couchbase 250114048 Mar 4 13:23 core.memcached.13679
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:23 core.memcached.13693
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:23 core.memcached.13707
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:23 core.memcached.13721
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:23 core.memcached.13735
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:24 core.memcached.13749
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:24 core.memcached.13766
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:24 core.memcached.13780
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:24 core.memcached.13946
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:24 core.memcached.13960
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:24 core.memcached.13974
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:25 core.memcached.13988
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:25 core.memcached.14003
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:25 core.memcached.14019
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:25 core.memcached.14033
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:25 core.memcached.14047
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:25 core.memcached.14061
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:26 core.memcached.14147
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:22 core.memcached.1438
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:26 core.memcached.14457
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:22 core.memcached.1452
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:22 core.memcached.1468
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:26 core.memcached.14787
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:23 core.memcached.1482
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:23 core.memcached.1497
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:26 core.memcached.14990
        rw------ 1 couchbase couchbase 250093568 Mar 4 13:26 core.memcached.15006
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:28 core.memcached.15020
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:28 core.memcached.15038
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:29 core.memcached.15052
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:29 core.memcached.15069
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:29 core.memcached.15083
        rw------ 1 couchbase couchbase 250097664 Mar 4 13:29 core.memcached.15097
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:23 core.memcached.1511
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:29 core.memcached.15111
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:29 core.memcached.15125
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:30 core.memcached.15139
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:30 core.memcached.15154
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:30 core.memcached.15170
        rw------ 1 couchbase couchbase 250109952 Mar 4 13:30 core.memcached.15184
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:30 core.memcached.15198
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:30 core.memcached.15212
        rw------ 1 couchbase couchbase 250093568 Mar 4 13:31 core.memcached.15226
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:31 core.memcached.15241
        rw------ 1 couchbase couchbase 250097664 Mar 4 12:23 core.memcached.1525
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:31 core.memcached.15255
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:31 core.memcached.15269
        rw------ 1 couchbase couchbase 250093568 Mar 4 13:31 core.memcached.15285
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:31 core.memcached.15299
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:32 core.memcached.15313
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:32 core.memcached.15328
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:32 core.memcached.15342
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:33 core.memcached.15358
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:33 core.memcached.15373
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:33 core.memcached.15387
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:23 core.memcached.1539
        rw------ 1 couchbase couchbase 250114048 Mar 4 13:33 core.memcached.15411
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:23 core.memcached.1553
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:33 core.memcached.15621
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:24 core.memcached.1567
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:24 core.memcached.1584
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:24 core.memcached.1598
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:33 core.memcached.16007
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:24 core.memcached.1612
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:24 core.memcached.1626
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:34 core.memcached.16327
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:24 core.memcached.1640
        rw------ 1 couchbase couchbase 250114048 Mar 4 13:34 core.memcached.16484
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:34 core.memcached.16498
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:34 core.memcached.16512
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:34 core.memcached.16526
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:25 core.memcached.1654
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:34 core.memcached.16540
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:35 core.memcached.16554
        rw------ 1 couchbase couchbase 101040128 Mar 4 13:35 core.memcached.16569
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:25 core.memcached.1669
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:25 core.memcached.1685
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:25 core.memcached.1699
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:25 core.memcached.1713
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:26 core.memcached.1727
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:26 core.memcached.1742
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:26 core.memcached.1756
        rw------ 1 couchbase couchbase 250097664 Mar 4 12:26 core.memcached.1770
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:26 core.memcached.1786
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:26 core.memcached.1800
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:27 core.memcached.1814
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:27 core.memcached.1829
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:27 core.memcached.1843
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:27 core.memcached.1857
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:27 core.memcached.2076
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:27 core.memcached.2434
        rw------ 1 couchbase couchbase 673476608 Mar 4 11:51 core.memcached.24610
        rw------ 1 couchbase couchbase 252207104 Mar 4 11:53 core.memcached.27971
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:54 core.memcached.27989
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:54 core.memcached.28008
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.28022
        rw------ 1 couchbase couchbase 251142144 Mar 4 11:55 core.memcached.28037
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.28053
        rw------ 1 couchbase couchbase 251158528 Mar 4 11:56 core.memcached.28067
        rw------ 1 couchbase couchbase 251146240 Mar 4 11:57 core.memcached.28084
        rw------ 1 couchbase couchbase 251158528 Mar 4 11:57 core.memcached.28099
        rw------ 1 couchbase couchbase 251158528 Mar 4 11:57 core.memcached.28113
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:58 core.memcached.28127
        rw------ 1 couchbase couchbase 251158528 Mar 4 11:59 core.memcached.28175
        rw------ 1 couchbase couchbase 250097664 Mar 4 12:00 core.memcached.28192
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:00 core.memcached.28207
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:00 core.memcached.28223
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:00 core.memcached.28237
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.28251
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.28279
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:01 core.memcached.28293
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.28307
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.28323
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.28337
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:02 core.memcached.28351
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:02 core.memcached.28366
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:28 core.memcached.2848
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:02 core.memcached.28759
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:02 core.memcached.29221
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:03 core.memcached.29434
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:03 core.memcached.29449
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:03 core.memcached.29463
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:03 core.memcached.29477
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:30 core.memcached.2948
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:03 core.memcached.29491
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:03 core.memcached.29505
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:04 core.memcached.29519
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:04 core.memcached.29536
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:04 core.memcached.29550
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:04 core.memcached.29564
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:04 core.memcached.29578
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:04 core.memcached.29592
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:05 core.memcached.29606
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:05 core.memcached.29621
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:05 core.memcached.29637
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:05 core.memcached.29651
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:05 core.memcached.29665
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:05 core.memcached.29679
        rw------ 1 couchbase couchbase 250097664 Mar 4 12:31 core.memcached.2968
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:06 core.memcached.29693
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:06 core.memcached.29708
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:06 core.memcached.29722
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:06 core.memcached.29736
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:06 core.memcached.29752
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:06 core.memcached.29766
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:07 core.memcached.29780
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:07 core.memcached.29795
        rw------ 1 couchbase couchbase 252211200 Mar 4 12:08 core.memcached.29809
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:32 core.memcached.2983
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:09 core.memcached.30010
        rw------ 1 couchbase couchbase 252211200 Mar 4 12:36 core.memcached.3002
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:09 core.memcached.30027
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:09 core.memcached.30041
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:09 core.memcached.30055
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:10 core.memcached.30069
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:36 core.memcached.3024
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:37 core.memcached.3042
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:10 core.memcached.30466
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:37 core.memcached.3057
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:21 core.memcached.307
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:37 core.memcached.3071
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:37 core.memcached.3085
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:10 core.memcached.30940
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:10 core.memcached.30954
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:10 core.memcached.30968
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:10 core.memcached.30982
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:37 core.memcached.3099
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:11 core.memcached.30996
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:11 core.memcached.31011
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:11 core.memcached.31025
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:11 core.memcached.31039
        rw------ 1 couchbase couchbase 252207104 Mar 4 12:12 core.memcached.31055
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:13 core.memcached.31072
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:13 core.memcached.31087
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:13 core.memcached.31101
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:13 core.memcached.31115
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:13 core.memcached.31129
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:13 core.memcached.31143
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:14 core.memcached.31157
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:37 core.memcached.3116
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:14 core.memcached.31174
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:14 core.memcached.31188
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:14 core.memcached.31202
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:15 core.memcached.31216
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:15 core.memcached.31231
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:15 core.memcached.31247
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:15 core.memcached.31261
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:15 core.memcached.31275
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:15 core.memcached.31289
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:38 core.memcached.3130
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.31303
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:16 core.memcached.31318
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.31332
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.31346
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:38 core.memcached.3145
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:38 core.memcached.3160
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.31683
        rw------ 1 couchbase couchbase 250114048 Mar 4 12:38 core.memcached.3174
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:38 core.memcached.3189
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:16 core.memcached.32045
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:38 core.memcached.3207
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:39 core.memcached.3226
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:21 core.memcached.323
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:17 core.memcached.32414
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:17 core.memcached.32429
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:17 core.memcached.32443
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:17 core.memcached.32457
        rw------ 1 couchbase couchbase 252211200 Mar 4 12:40 core.memcached.3247
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:17 core.memcached.32473
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:18 core.memcached.32487
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:18 core.memcached.32502
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:18 core.memcached.32516
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:18 core.memcached.32530
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:18 core.memcached.32544
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:18 core.memcached.32558
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:19 core.memcached.32572
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:19 core.memcached.32589
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:19 core.memcached.32603
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:19 core.memcached.32617
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:19 core.memcached.32631
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:19 core.memcached.32645
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:20 core.memcached.32659
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:41 core.memcached.3266
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:20 core.memcached.32676
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:20 core.memcached.32690
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:20 core.memcached.32704
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:20 core.memcached.32718
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:21 core.memcached.32732
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:21 core.memcached.32747
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:21 core.memcached.32761
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:42 core.memcached.3281
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:21 core.memcached.343
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:42 core.memcached.3433
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:22 core.memcached.357
        rw------ 1 couchbase couchbase 250114048 Mar 4 12:42 core.memcached.3780
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:42 core.memcached.4028
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:42 core.memcached.4427
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:42 core.memcached.4447
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:43 core.memcached.4461
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:43 core.memcached.4476
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:43 core.memcached.4490
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:43 core.memcached.4504
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:43 core.memcached.4518
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:43 core.memcached.4532
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:44 core.memcached.4546
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:44 core.memcached.4563
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:44 core.memcached.4577
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:44 core.memcached.4591
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:44 core.memcached.4605
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:44 core.memcached.4619
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:45 core.memcached.4633
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:45 core.memcached.4648
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:45 core.memcached.4664
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:45 core.memcached.4678
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:45 core.memcached.4692
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:45 core.memcached.4706
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:46 core.memcached.4720
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:46 core.memcached.4735
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:48 core.memcached.4751
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:48 core.memcached.4769
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:48 core.memcached.4783
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:48 core.memcached.4797
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:49 core.memcached.4811
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:49 core.memcached.4828
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:49 core.memcached.4842
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:49 core.memcached.4856
        rw------ 1 couchbase couchbase 250093568 Mar 4 12:49 core.memcached.4870
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:49 core.memcached.5121
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:50 core.memcached.5386
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:50 core.memcached.5619
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:50 core.memcached.5908
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:50 core.memcached.5967
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:50 core.memcached.5981
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:50 core.memcached.5995
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:51 core.memcached.6009
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:51 core.memcached.6024
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:51 core.memcached.6038
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:51 core.memcached.6052
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:51 core.memcached.6068
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:51 core.memcached.6082
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:52 core.memcached.6096
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:52 core.memcached.6111
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:52 core.memcached.6125
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:52 core.memcached.6139
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:52 core.memcached.6153
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:52 core.memcached.6169
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:53 core.memcached.6183
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:53 core.memcached.6198
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:53 core.memcached.6212
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:53 core.memcached.6226
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:53 core.memcached.6240
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:53 core.memcached.6254
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:54 core.memcached.6268
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:54 core.memcached.6285
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:54 core.memcached.6299
        rw------ 1 couchbase couchbase 250093568 Mar 4 12:54 core.memcached.6313
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:54 core.memcached.6327
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:54 core.memcached.6343
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:55 core.memcached.6357
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:55 core.memcached.6372
        rw------ 1 couchbase couchbase 250114048 Mar 4 12:55 core.memcached.6388
        rw------ 1 couchbase couchbase 250114048 Mar 4 12:55 core.memcached.6585
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:22 core.memcached.668
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:55 core.memcached.6825
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:55 core.memcached.7162
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:56 core.memcached.7366
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:56 core.memcached.7381
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:56 core.memcached.7395
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:56 core.memcached.7409
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:56 core.memcached.7425
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:56 core.memcached.7439
        rw------ 1 couchbase couchbase 250114048 Mar 4 12:57 core.memcached.7453
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:57 core.memcached.7468
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:57 core.memcached.7482
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:57 core.memcached.7496
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:57 core.memcached.7510
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:57 core.memcached.7526
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:58 core.memcached.7540
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:58 core.memcached.7555
        rw------ 1 couchbase couchbase 250114048 Mar 4 12:58 core.memcached.7569
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:58 core.memcached.7583
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:58 core.memcached.7628
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:58 core.memcached.7642
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:59 core.memcached.7656
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:59 core.memcached.7673
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:59 core.memcached.7687
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:59 core.memcached.7701
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:59 core.memcached.7715
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:59 core.memcached.7729
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:00 core.memcached.7743
        rw------ 1 couchbase couchbase 250097664 Mar 4 13:00 core.memcached.7758
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:00 core.memcached.7774
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:00 core.memcached.7788
        rw------ 1 couchbase couchbase 250097664 Mar 4 13:00 core.memcached.7802
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:01 core.memcached.7816
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:01 core.memcached.8196
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:01 core.memcached.8418
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:01 core.memcached.8734
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:01 core.memcached.8912
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:01 core.memcached.8926
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:02 core.memcached.8940
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:02 core.memcached.8955
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:02 core.memcached.8969
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:02 core.memcached.8983
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:02 core.memcached.8997
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:02 core.memcached.9013
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:03 core.memcached.9027
        rw------ 1 couchbase couchbase 250109952 Mar 4 13:03 core.memcached.9042
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:03 core.memcached.9056
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:03 core.memcached.9070
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:03 core.memcached.9084
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:03 core.memcached.9098
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:04 core.memcached.9112
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:04 core.memcached.9129
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:04 core.memcached.9143
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:04 core.memcached.9157
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:04 core.memcached.9171
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:04 core.memcached.9185
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:05 core.memcached.9199
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:05 core.memcached.9214
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:05 core.memcached.9230
        rw------ 1 couchbase couchbase 250109952 Mar 4 13:05 core.memcached.9244
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:05 core.memcached.9258
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:05 core.memcached.9272
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:06 core.memcached.9286
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:06 core.memcached.9301
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:06 core.memcached.9315
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:06 core.memcached.9329
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:06 core.memcached.9464
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:06 core.memcached.9703
        rw-rr- 1 jenkins jenkins 126761566 Mar 5 00:40 couchbase-server-enterprise_x86_64_2.0.2-728-rel.rpm
        drwxrwxrwt 2 couchbase couchbase 4096 Oct 12 06:54 .font-unix
        srw-rw-rw- 1 couchbase couchbase 0 Oct 12 06:54 .gdm_socket
        drwxrwxrwt 2 couchbase couchbase 4096 Oct 12 06:54 .ICE-unix
        drwxrwxrwt 2 couchbase couchbase 4096 Oct 12 06:54 .X11-unix

        when I removed only a small part of them I got usage 95%
        This means that after the first сrash, there should not be a problem with disk space

        [jenkins@cen-1709 tmp]$ sudo rm -rf core.memcached.9*
        [jenkins@cen-1709 tmp]$ df
        Filesystem 1K-blocks Used Available Use% Mounted on
        /dev/mapper/VolGroup00-LogVol00
        55007284 49218324 2949616 95% /
        /dev/sda1 101086 19983 75884 21% /boot
        tmpfs 2021844 0 2021844 0% /dev/shm

        on the second node 10.5.2.14 disk space is OK, but also there are many crashes:

        [jenkins@cen-1710 tmp]$ ls -la
        total 27881444
        drwxrwxrwt 5 couchbase couchbase 24576 Mar 5 02:04 .
        drwxr-xr-x 25 root root 4096 Feb 18 03:09 ..
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:09 core.memcached.10168
        rw------ 1 couchbase couchbase 250097664 Mar 4 12:09 core.memcached.10220
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:09 core.memcached.10234
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:09 core.memcached.10248
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:09 core.memcached.10262
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:10 core.memcached.10278
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:11 core.memcached.10292
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:11 core.memcached.10307
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:11 core.memcached.10324
        rw------ 1 couchbase couchbase 250093568 Mar 4 12:11 core.memcached.10338
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:12 core.memcached.10352
        rw------ 1 couchbase couchbase 250097664 Mar 4 12:12 core.memcached.10366
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:12 core.memcached.10381
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:12 core.memcached.10397
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:12 core.memcached.10411
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:13 core.memcached.10425
        rw------ 1 couchbase couchbase 250093568 Mar 4 12:14 core.memcached.10442
        rw------ 1 couchbase couchbase 250093568 Mar 4 12:14 core.memcached.10456
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:14 core.memcached.10471
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:14 core.memcached.10485
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:14 core.memcached.10499
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:15 core.memcached.10513
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.10530
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.10544
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:16 core.memcached.10561
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.10575
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:16 core.memcached.10589
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:16 core.memcached.10666
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:17 core.memcached.11045
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:17 core.memcached.11388
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:17 core.memcached.11670
        rw------ 1 couchbase couchbase 250093568 Mar 4 12:17 core.memcached.11686
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:19 core.memcached.11700
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:20 core.memcached.11749
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:20 core.memcached.11763
        rw------ 1 couchbase couchbase 250093568 Mar 4 12:20 core.memcached.11778
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:20 core.memcached.11792
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:20 core.memcached.11806
        rw------ 1 couchbase couchbase 252207104 Mar 4 12:23 core.memcached.11820
        rw------ 1 couchbase couchbase 252207104 Mar 4 12:25 core.memcached.11843
        rw------ 1 couchbase couchbase 252207104 Mar 4 12:28 core.memcached.11861
        rw------ 1 couchbase couchbase 250093568 Mar 4 12:28 core.memcached.11882
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:28 core.memcached.11896
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:28 core.memcached.11910
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:28 core.memcached.11926
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:29 core.memcached.11940
        rw------ 1 couchbase couchbase 250097664 Mar 4 12:29 core.memcached.11954
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:29 core.memcached.11969
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:29 core.memcached.11983
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:29 core.memcached.11997
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:29 core.memcached.12011
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:30 core.memcached.12027
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:30 core.memcached.12041
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:30 core.memcached.12056
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:30 core.memcached.12070
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:30 core.memcached.12084
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:31 core.memcached.12116
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:31 core.memcached.12779
        rw------ 1 couchbase couchbase 250114048 Mar 4 12:31 core.memcached.13085
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:31 core.memcached.13167
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:31 core.memcached.13181
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:31 core.memcached.13195
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:32 core.memcached.13209
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:32 core.memcached.13223
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:32 core.memcached.13238
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:32 core.memcached.13254
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:32 core.memcached.13268
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:32 core.memcached.13282
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:33 core.memcached.13296
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:33 core.memcached.13310
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:33 core.memcached.13325
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:33 core.memcached.13339
        rw------ 1 couchbase couchbase 250089472 Mar 4 12:33 core.memcached.13353
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:33 core.memcached.13369
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:34 core.memcached.13383
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:34 core.memcached.13397
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:34 core.memcached.13412
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:34 core.memcached.13571
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:34 core.memcached.13590
        rw------ 1 couchbase couchbase 250093568 Mar 4 12:34 core.memcached.13604
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:35 core.memcached.13620
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:35 core.memcached.13634
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:35 core.memcached.13649
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:35 core.memcached.13663
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:35 core.memcached.13677
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:35 core.memcached.13691
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:36 core.memcached.13705
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:36 core.memcached.13719
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:36 core.memcached.13736
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:36 core.memcached.13983
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:36 core.memcached.14322
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:39 core.memcached.14652
        rw------ 1 couchbase couchbase 250093568 Mar 4 12:39 core.memcached.14672
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:39 core.memcached.14687
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:39 core.memcached.14701
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:39 core.memcached.14715
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:39 core.memcached.14729
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:40 core.memcached.14745
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:40 core.memcached.14759
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:40 core.memcached.14774
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:40 core.memcached.14788
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:40 core.memcached.14802
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:40 core.memcached.14816
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:41 core.memcached.14980
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:41 core.memcached.14994
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:41 core.memcached.15011
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:41 core.memcached.15025
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:41 core.memcached.15039
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:41 core.memcached.15053
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:42 core.memcached.15067
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:42 core.memcached.15081
        rw------ 1 couchbase couchbase 252194816 Mar 4 12:46 core.memcached.15096
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:46 core.memcached.15122
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:46 core.memcached.15136
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:46 core.memcached.15150
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:46 core.memcached.15164
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:47 core.memcached.15178
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:47 core.memcached.15192
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:47 core.memcached.15207
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:47 core.memcached.15223
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:47 core.memcached.15237
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:47 core.memcached.15251
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:48 core.memcached.15383
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:48 core.memcached.15781
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:48 core.memcached.16246
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:49 core.memcached.16332
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:49 core.memcached.16348
        rw------ 1 couchbase couchbase 252211200 Mar 4 12:51 core.memcached.16363
        rw------ 1 couchbase couchbase 252194816 Mar 4 12:53 core.memcached.16383
        rw------ 1 couchbase couchbase 252207104 Mar 4 12:59 core.memcached.16403
        rw------ 1 couchbase couchbase 252207104 Mar 4 13:00 core.memcached.16431
        rw------ 1 couchbase couchbase 250109952 Mar 4 13:01 core.memcached.16448
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:01 core.memcached.16475
        rw------ 1 couchbase couchbase 252207104 Mar 4 13:05 core.memcached.16492
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:05 core.memcached.16518
        rw------ 1 couchbase couchbase 250109952 Mar 4 13:05 core.memcached.16564
        rw------ 1 couchbase couchbase 252207104 Mar 4 13:07 core.memcached.16578
        rw------ 1 couchbase couchbase 250093568 Mar 4 13:08 core.memcached.16598
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:08 core.memcached.16612
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:08 core.memcached.16627
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:08 core.memcached.16641
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:08 core.memcached.16655
        rw------ 1 couchbase couchbase 250114048 Mar 4 13:08 core.memcached.16671
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:09 core.memcached.16685
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:10 core.memcached.16699
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:10 core.memcached.17164
        rw------ 1 couchbase couchbase 252190720 Mar 4 13:19 core.memcached.17475
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:20 core.memcached.17794
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:20 core.memcached.17808
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:20 core.memcached.17822
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:20 core.memcached.17837
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:20 core.memcached.17851
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:20 core.memcached.17865
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:21 core.memcached.17879
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:21 core.memcached.17893
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:21 core.memcached.17909
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:21 core.memcached.17924
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:21 core.memcached.17938
        rw------ 1 couchbase couchbase 252207104 Mar 4 13:24 core.memcached.17952
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:25 core.memcached.17972
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:26 core.memcached.17990
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:26 core.memcached.18007
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:27 core.memcached.18021
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:27 core.memcached.18188
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:27 core.memcached.18202
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:27 core.memcached.18219
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:27 core.memcached.18233
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:27 core.memcached.18247
        rw------ 1 couchbase couchbase 250114048 Mar 4 13:28 core.memcached.18261
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:28 core.memcached.18275
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:28 core.memcached.18289
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:28 core.memcached.18304
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:28 core.memcached.18649
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:28 core.memcached.18980
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:29 core.memcached.19219
        rw------ 1 couchbase couchbase 251162624 Mar 4 13:30 core.memcached.19233
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:30 core.memcached.19250
        rw------ 1 couchbase couchbase 251146240 Mar 4 13:31 core.memcached.19265
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:32 core.memcached.19282
        rw------ 1 couchbase couchbase 250114048 Mar 4 13:32 core.memcached.19299
        rw------ 1 couchbase couchbase 250114048 Mar 4 13:32 core.memcached.19313
        rw------ 1 couchbase couchbase 252207104 Mar 4 13:34 core.memcached.19327
        rw------ 1 couchbase couchbase 251158528 Mar 4 13:35 core.memcached.19497
        rw------ 1 couchbase couchbase 251142144 Mar 4 13:35 core.memcached.19513
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:44 core.memcached.6300
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:45 core.memcached.6328
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:45 core.memcached.6342
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:45 core.memcached.6357
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:45 core.memcached.6371
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:45 core.memcached.6385
        rw------ 1 couchbase couchbase 250097664 Mar 4 11:45 core.memcached.6399
        rw------ 1 couchbase couchbase 250097664 Mar 4 11:46 core.memcached.6413
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:46 core.memcached.6427
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:46 core.memcached.6444
        rw------ 1 couchbase couchbase 250097664 Mar 4 11:46 core.memcached.6458
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:46 core.memcached.6472
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:46 core.memcached.6486
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:47 core.memcached.6500
        rw------ 1 couchbase couchbase 250093568 Mar 4 11:47 core.memcached.6514
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:47 core.memcached.6529
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:47 core.memcached.6545
        rw------ 1 couchbase couchbase 250097664 Mar 4 11:47 core.memcached.6559
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:47 core.memcached.6573
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:48 core.memcached.6587
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:48 core.memcached.6601
        rw------ 1 couchbase couchbase 250097664 Mar 4 11:48 core.memcached.6616
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:48 core.memcached.6630
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:48 core.memcached.6644
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:48 core.memcached.6686
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:49 core.memcached.6700
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:49 core.memcached.6714
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:49 core.memcached.6729
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:49 core.memcached.6743
        rw------ 1 couchbase couchbase 250093568 Mar 4 11:49 core.memcached.6757
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:49 core.memcached.6771
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:50 core.memcached.6787
        rw------ 1 couchbase couchbase 250093568 Mar 4 11:50 core.memcached.6801
        rw------ 1 couchbase couchbase 250093568 Mar 4 11:50 core.memcached.6816
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:50 core.memcached.6830
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:50 core.memcached.6844
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:50 core.memcached.6858
        rw------ 1 couchbase couchbase 250097664 Mar 4 11:51 core.memcached.6872
        rw------ 1 couchbase couchbase 250093568 Mar 4 11:51 core.memcached.6886
        rw------ 1 couchbase couchbase 250093568 Mar 4 11:51 core.memcached.6903
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:51 core.memcached.6917
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:51 core.memcached.6931
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:51 core.memcached.6945
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:52 core.memcached.6959
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:52 core.memcached.6973
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:52 core.memcached.6988
        rw------ 1 couchbase couchbase 250097664 Mar 4 11:52 core.memcached.7004
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:52 core.memcached.7018
        rw------ 1 couchbase couchbase 250093568 Mar 4 11:52 core.memcached.7032
        rw------ 1 couchbase couchbase 250093568 Mar 4 11:53 core.memcached.7046
        rw------ 1 couchbase couchbase 250097664 Mar 4 11:53 core.memcached.7060
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:53 core.memcached.7075
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:53 core.memcached.7089
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:53 core.memcached.7103
        rw------ 1 couchbase couchbase 250097664 Mar 4 11:53 core.memcached.7119
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:54 core.memcached.7133
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:54 core.memcached.7147
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:54 core.memcached.7162
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:54 core.memcached.7176
        rw------ 1 couchbase couchbase 250114048 Mar 4 11:54 core.memcached.7190
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.7204
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.7220
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.7235
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.7249
        rw------ 1 couchbase couchbase 251142144 Mar 4 11:55 core.memcached.7263
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.7277
        rw------ 1 couchbase couchbase 251146240 Mar 4 11:56 core.memcached.7291
        rw------ 1 couchbase couchbase 251158528 Mar 4 11:56 core.memcached.7305
        rw------ 1 couchbase couchbase 251158528 Mar 4 11:56 core.memcached.7322
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:56 core.memcached.7336
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:56 core.memcached.7350
        rw------ 1 couchbase couchbase 251158528 Mar 4 11:57 core.memcached.7364
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:57 core.memcached.7378
        rw------ 1 couchbase couchbase 251158528 Mar 4 11:57 core.memcached.7393
        rw------ 1 couchbase couchbase 251158528 Mar 4 11:57 core.memcached.7409
        rw------ 1 couchbase couchbase 251146240 Mar 4 11:57 core.memcached.7423
        rw------ 1 couchbase couchbase 251158528 Mar 4 11:57 core.memcached.7437
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:58 core.memcached.7451
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:58 core.memcached.7465
        rw------ 1 couchbase couchbase 251158528 Mar 4 11:58 core.memcached.7565
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:58 core.memcached.7701
        rw------ 1 couchbase couchbase 251158528 Mar 4 11:58 core.memcached.7715
        rw------ 1 couchbase couchbase 251142144 Mar 4 11:59 core.memcached.7731
        rw------ 1 couchbase couchbase 250109952 Mar 4 11:59 core.memcached.7745
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:59 core.memcached.7760
        rw------ 1 couchbase couchbase 251146240 Mar 4 11:59 core.memcached.7774
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:59 core.memcached.7788
        rw------ 1 couchbase couchbase 251162624 Mar 4 11:59 core.memcached.7802
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:00 core.memcached.7818
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:00 core.memcached.7832
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:00 core.memcached.7947
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:00 core.memcached.8405
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:00 core.memcached.8692
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:01 core.memcached.8706
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.8736
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:01 core.memcached.8750
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.8766
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:02 core.memcached.8780
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:02 core.memcached.8794
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:02 core.memcached.8809
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:02 core.memcached.8825
        rw------ 1 couchbase couchbase 252207104 Mar 4 12:04 core.memcached.8839
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:05 core.memcached.8857
        rw------ 1 couchbase couchbase 250109952 Mar 4 12:06 core.memcached.8906
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:06 core.memcached.8920
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:06 core.memcached.8937
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:06 core.memcached.8951
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:06 core.memcached.8965
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:06 core.memcached.8979
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:07 core.memcached.8993
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:07 core.memcached.9007
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:07 core.memcached.9022
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:07 core.memcached.9038
        rw------ 1 couchbase couchbase 250093568 Mar 4 12:07 core.memcached.9052
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:07 core.memcached.9066
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:08 core.memcached.9080
        rw------ 1 couchbase couchbase 250114048 Mar 4 12:08 core.memcached.9094
        rw------ 1 couchbase couchbase 251162624 Mar 4 12:08 core.memcached.9109
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:08 core.memcached.9123
        rw------ 1 couchbase couchbase 251146240 Mar 4 12:08 core.memcached.9137
        rw------ 1 couchbase couchbase 251158528 Mar 4 12:08 core.memcached.9262
        rw------ 1 couchbase couchbase 251142144 Mar 4 12:09 core.memcached.9740
        rw-rr- 1 jenkins jenkins 126761566 Mar 5 00:40 couchbase-server-enterprise_x86_64_2.0.2-728-rel.rpm
        drwxrwxrwt 2 couchbase couchbase 4096 Oct 11 08:01 .font-unix
        srw-rw-rw- 1 couchbase couchbase 0 Oct 11 08:01 .gdm_socket
        drwxrwxrwt 2 couchbase couchbase 4096 Oct 11 08:01 .ICE-unix
        drwxrwxrwt 2 couchbase couchbase 4096 Oct 11 08:01 .X11-unix
        [jenkins@cen-1710 tmp]$ df
        Filesystem 1K-blocks Used Available Use% Mounted on
        /dev/mapper/VolGroup00-LogVol00
        55007284 36258700 15909240 70% /
        /dev/sda1 101086 19983 75884 21% /boot
        tmpfs 2021844 0 2021844 0% /dev/shm

        [jenkins@cen-1709 tmp]$ sudo rm -rf core.memcached.9*
        [jenkins@cen-1709 tmp]$ df
        Filesystem 1K-blocks Used Available Use% Mounted on
        /dev/mapper/VolGroup00-LogVol00
        55007284 49218324 2949616 95% /
        /dev/sda1 101086 19983 75884 21% /boot
        tmpfs 2021844 0 2021844 0% /dev/shm
        [jenkins@cen-1709 tmp]$ sudo gdb /opt/couchbase/bin/memcached core.memcached.24610
        GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
        Copyright (C) 2009 Free Software Foundation, Inc.
        License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
        This is free software: you are free to change and redistribute it.
        There is NO WARRANTY, to the extent permitted by law. Type "show copying"
        and "show warranty" for details.
        This GDB was configured as "x86_64-redhat-linux-gnu".
        For bug reporting instructions, please see:
        <http://www.gnu.org/software/gdb/bugs/>...
        Reading symbols from /opt/couchbase/bin/memcached...done.

        warning: exec file is newer than core file.
        [New Thread 24624]
        [New Thread 27925]
        [New Thread 27924]
        [New Thread 27923]
        [New Thread 27922]
        [New Thread 27921]
        [New Thread 24626]
        [New Thread 24625]
        [New Thread 24623]
        [New Thread 24622]
        [New Thread 24621]
        [New Thread 24619]
        [New Thread 24618]
        [New Thread 24610]
        Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
        Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
        Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
        Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
        Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libdl.so.2
        Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libm.so.6
        Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
        Loaded symbols for /lib64/librt.so.1
        Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
        Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
        Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
        [Thread debugging using libthread_db enabled]
        Loaded symbols for /lib64/libpthread.so.0
        Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libc.so.6
        Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
        Loaded symbols for /lib64/ld-linux-x86-64.so.2
        Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
        Loaded symbols for /usr/lib64/libstdc++.so.6
        Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libgcc_s.so.1
        Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
        Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
        Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
        Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
        Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
        Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
        Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
        Loaded symbols for /opt/couchbase/lib/memcached/ep.so
        Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
        Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
        Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
        Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
        Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
        Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
        Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
        Loaded symbols for /opt/couchbase/lib/libicudata.so.44
        Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
        Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
        Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libnss_files.so.2

        warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff39fac000
        Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
        Program terminated with signal 11, Segmentation fault.
        #0 0x00002aaaaaf0eb1a in LOG (severity=<value optimized out>, fmt=<value optimized out>) at src/ep_engine.cc:1184
        1184 src/ep_engine.cc: No such file or directory.
        in src/ep_engine.cc
        (gdb) t a a bt

        Thread 14 (Thread 0x2baa8c1f7220 (LWP 24610)):
        #0 0x0000003f142d48a8 in epoll_wait () from /lib64/libc.so.6
        #1 0x00002baa8bd74576 in epoll_dispatch (base=0xc916000, tv=<value optimized out>) at epoll.c:404
        #2 0x00002baa8bd5fe44 in event_base_loop (base=0xc916000, flags=<value optimized out>) at event.c:1558
        #3 0x0000000000409742 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7918

        Thread 13 (Thread 24618):
        #0 0x0000003f142c678b in read () from /lib64/libc.so.6
        #1 0x0000003f1426cd57 in _IO_new_file_underflow () from /lib64/libc.so.6
        #2 0x0000003f1426d71e in _IO_default_uflow_internal () from /lib64/libc.so.6
        #3 0x0000003f14262804 in _IO_getline_info_internal () from /lib64/libc.so.6
        #4 0x0000003f142616a9 in fgets () from /lib64/libc.so.6
        #5 0x00002baa8c1f8939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
        #6 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003f142d44bd in clone () from /lib64/libc.so.6

        Thread 12 (Thread 24619):
        #0 0x0000003f14e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x00002aaaaaaae4d6 in logger_thead_main (arg=0x80b2040) at extensions/loggers/file_logger.c:368
        #2 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0
        #3 0x0000003f142d44bd in clone () from /lib64/libc.so.6

        Thread 11 (Thread 24621):
        #0 0x0000003f142d48a8 in epoll_wait () from /lib64/libc.so.6
        #1 0x00002baa8bd74576 in epoll_dispatch (base=0xc916500, tv=<value optimized out>) at epoll.c:404
        #2 0x00002baa8bd5fe44 in event_base_loop (base=0xc916500, flags=<value optimized out>) at event.c:1558
        #3 0x0000000000414504 in worker_libevent (arg=0x80b5900) at daemon/thread.c:301
        #4 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0
        #5 0x0000003f142d44bd in clone () from /lib64/libc.so.6

        Thread 10 (Thread 24622):
        #0 0x0000003f142d48a8 in epoll_wait () from /lib64/libc.so.6
        #1 0x00002baa8bd74576 in epoll_dispatch (base=0xc916280, tv=<value optimized out>) at epoll.c:404
        #2 0x00002baa8bd5fe44 in event_base_loop (base=0xc916280, flags=<value optimized out>) at event.c:1558
        #3 0x0000000000414504 in worker_libevent (arg=0x80b59f8) at daemon/thread.c:301
        #4 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0
        #5 0x0000003f142d44bd in clone () from /lib64/libc.so.6

        Thread 9 (Thread 24623):
        #0 0x0000003f142d48a8 in epoll_wait () from /lib64/libc.so.6
        #1 0x00002baa8bd74576 in epoll_dispatch (base=0xc916c80, tv=<value optimized out>) at epoll.c:404
        #2 0x00002baa8bd5fe44 in event_base_loop (base=0xc916c80, flags=<value optimized out>) at event.c:1558
        #3 0x0000000000414504 in worker_libevent (arg=0x80b5af0) at daemon/thread.c:301
        #4 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0
        #5 0x0000003f142d44bd in clone () from /lib64/libc.so.6

        Thread 8 (Thread 24625):
        #0 0x0000003f142d48a8 in epoll_wait () from /lib64/libc.so.6
        #1 0x00002baa8bd74576 in epoll_dispatch (base=0xc916780, tv=<value optimized out>) at epoll.c:404
        #2 0x00002baa8bd5fe44 in event_base_loop (base=0xc916780, flags=<value optimized out>) at event.c:1558
        #3 0x0000000000414504 in worker_libevent (arg=0x80b5ce0) at daemon/thread.c:301
        #4 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0
        #5 0x0000003f142d44bd in clone () from /lib64/libc.so.6

        --Type <return> to continue, or q <return> to quit--
        Thread 7 (Thread 24626):
        #0 0x0000003f1429a541 in nanosleep () from /lib64/libc.so.6
        #1 0x0000003f142cded4 in usleep () from /lib64/libc.so.6
        #2 0x00002aaaaaf30255 in updateStatsThread (arg=0x80b24c0) at src/memory_tracker.cc:31
        #3 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0
        #4 0x0000003f142d44bd in clone () from /lib64/libc.so.6

        Thread 6 (Thread 27921):
        #0 0x0000003f14e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x00002aaaaaef2e58 in wait (this=0xc963440, d=...) at src/syncobject.hh:57
        #2 IdleTask::run (this=0xc963440, d=...) at src/dispatcher.cc:328
        #3 0x00002aaaaaef568a in Dispatcher::run (this=0x154d6a80) at src/dispatcher.cc:171
        #4 0x00002aaaaaef5f3d in launch_dispatcher_thread (arg=0x154d6a80) at src/dispatcher.cc:28
        #5 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0
        #6 0x0000003f142d44bd in clone () from /lib64/libc.so.6

        Thread 5 (Thread 27922):
        #0 0x0000003f14e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x00002aaaaaef2e58 in wait (this=0xc9637a0, d=...) at src/syncobject.hh:57
        #2 IdleTask::run (this=0xc9637a0, d=...) at src/dispatcher.cc:328
        #3 0x00002aaaaaef568a in Dispatcher::run (this=0x154d68c0) at src/dispatcher.cc:171
        #4 0x00002aaaaaef5f3d in launch_dispatcher_thread (arg=0x154d68c0) at src/dispatcher.cc:28
        #5 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0
        #6 0x0000003f142d44bd in clone () from /lib64/libc.so.6

        Thread 4 (Thread 27923):
        #0 0x0000003f14e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x00002aaaaaef2e58 in wait (this=0xc962c60, d=...) at src/syncobject.hh:57
        #2 IdleTask::run (this=0xc962c60, d=...) at src/dispatcher.cc:328
        #3 0x00002aaaaaef568a in Dispatcher::run (this=0x154d6700) at src/dispatcher.cc:171
        #4 0x00002aaaaaef5f3d in launch_dispatcher_thread (arg=0x154d6700) at src/dispatcher.cc:28
        #5 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0
        #6 0x0000003f142d44bd in clone () from /lib64/libc.so.6

        Thread 3 (Thread 27924):
        #0 0x0000003f14e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x00002aaaaaef2e58 in wait (this=0xc9633b0, d=...) at src/syncobject.hh:57
        #2 IdleTask::run (this=0xc9633b0, d=...) at src/dispatcher.cc:328
        #3 0x00002aaaaaef568a in Dispatcher::run (this=0x154d6540) at src/dispatcher.cc:171
        #4 0x00002aaaaaef5f3d in launch_dispatcher_thread (arg=0x154d6540) at src/dispatcher.cc:28
        #5 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0
        #6 0x0000003f142d44bd in clone () from /lib64/libc.so.6

        Thread 2 (Thread 27925):
        #0 0x0000003f14e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x00002aaaaaf1028f in wait (this=0x14550900) at src/syncobject.hh:57
        #2 wait (this=0x14550900) at src/syncobject.hh:73
        #3 wait (this=0x14550900) at src/tapconnmap.hh:169
        #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x14550900) at src/ep_engine.cc:3406
        #5 0x00002aaaaaf10373 in EvpNotifyPendingConns (arg=0x14550900) at src/ep_engine.cc:1139
        #6 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003f142d44bd in clone () from /lib64/libc.so.6

        Thread 1 (Thread 0x4477a940 (LWP 24624)):
        #0 0x00002aaaaaf0eb1a in LOG (severity=<value optimized out>, fmt=<value optimized out>) at src/ep_engine.cc:1184
        --Type <return> to continue, or q <return> to quit--
        #1 0x34202c343634202c in ?? ()
        #2 0x2c363634202c3536 in ?? ()
        #3 0x3634202c37363420 in ?? ()
        #4 0x202c393634202c38 in ?? ()
        #5 0x313734202c303734 in ?? ()
        #6 0x34202c323734202c in ?? ()
        #7 0x2c343734202c3337 in ?? ()
        #8 0x3734202c35373420 in ?? ()
        #9 0x202c373734202c36 in ?? ()
        #10 0x393734202c383734 in ?? ()
        #11 0x34202c303834202c in ?? ()
        #12 0x2c323834202c3138 in ?? ()
        #13 0x3834202c33383420 in ?? ()
        #14 0x202c353834202c34 in ?? ()
        #15 0x373834202c363834 in ?? ()
        #16 0x34202c383834202c in ?? ()
        #17 0x2c303934202c3938 in ?? ()
        #18 0x3934202c31393420 in ?? ()
        #19 0x202c333934202c32 in ?? ()
        #20 0x353934202c343934 in ?? ()
        #21 0x34202c363934202c in ?? ()
        #22 0x2c383934202c3739 in ?? ()
        #23 0x3035202c39393420 in ?? ()
        #24 0x202c313035202c30 in ?? ()
        #25 0x333035202c323035 in ?? ()
        #26 0x35202c343035202c in ?? ()
        #27 0x2c363035202c3530 in ?? ()
        #28 0x3035202c37303520 in ?? ()
        #29 0x202c393035202c38 in ?? ()
        #30 0x313135202c303135 in ?? ()
        #31 0x00000000000a202c in ?? ()
        #32 0x0000000000000000 in ?? ()

        don't have collect info due to out of space on slave :

        IMPOSSIBLE TO GRAB CBCOLLECT FROM 10.1.3.118
        Traceback (most recent call last):
        File "testrunner", line 362, in <module>
        xunit.write("

        {0}

        /report-

        {1}

        ".format(logs_folder, str_time))
        File "lib/xunit.py", line 102, in write
        report_xml_file.close()
        IOError: [Errno 28] No space left on device

        core dumps:
        jenkins(QA keys)@10.5.2.14:/tmp/backup/MB-7652_core.memcached.6300
        jenkins(QA keys)@10.5.2.13:/tmp/backup/MB-7652_core.memcached.24610

        Mike, whether the lack of disk space causes crashes?

        Show
        andreibaranouski Andrei Baranouski added a comment - Mike, I got the same behaviour and have a question: yes, indeed on one node was out of space on disk when I checked [jenkins@cen-1709 ~] $ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 55007284 51424616 743324 99% / /dev/sda1 101086 19983 75884 21% /boot tmpfs 2021844 0 2021844 0% /dev/shm but this was achieved when the memcached started to generate a lot of core dumps: 10.5.2.13 [jenkins@cen-1709 tmp] $ ls -la total 41887784 drwxrwxrwt 5 couchbase couchbase 20480 Mar 5 02:17 . drwxr-xr-x 25 root root 4096 Feb 18 03:09 .. rw ------ 1 couchbase couchbase 251162624 Mar 4 13:07 core.memcached.10010 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:07 core.memcached.10369 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:07 core.memcached.10426 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:07 core.memcached.10440 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:07 core.memcached.10454 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:08 core.memcached.10470 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:09 core.memcached.10485 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:09 core.memcached.10502 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:09 core.memcached.10516 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:09 core.memcached.10530 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:09 core.memcached.10544 rw ------ 1 couchbase couchbase 250097664 Mar 4 13:09 core.memcached.10558 rw ------ 1 couchbase couchbase 250093568 Mar 4 13:10 core.memcached.10572 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:10 core.memcached.10587 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:10 core.memcached.10603 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:10 core.memcached.10617 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:10 core.memcached.10631 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:10 core.memcached.10645 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:11 core.memcached.10659 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:11 core.memcached.10674 rw ------ 1 couchbase couchbase 250114048 Mar 4 13:11 core.memcached.10688 rw ------ 1 couchbase couchbase 250109952 Mar 4 13:11 core.memcached.10702 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:11 core.memcached.10718 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:11 core.memcached.10732 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:12 core.memcached.10746 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:22 core.memcached.1076 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:12 core.memcached.10761 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:12 core.memcached.10775 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:12 core.memcached.10789 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:12 core.memcached.10803 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:12 core.memcached.10819 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:13 core.memcached.10833 rw ------ 1 couchbase couchbase 250114048 Mar 4 13:13 core.memcached.10848 rw ------ 1 couchbase couchbase 250097664 Mar 4 13:13 core.memcached.11068 rw ------ 1 couchbase couchbase 250097664 Mar 4 13:13 core.memcached.11293 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:13 core.memcached.11546 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:13 core.memcached.11911 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:14 core.memcached.11942 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:14 core.memcached.11959 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:14 core.memcached.11973 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:14 core.memcached.11987 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:14 core.memcached.12001 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:14 core.memcached.12016 rw ------ 1 couchbase couchbase 250093568 Mar 4 13:15 core.memcached.12030 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:15 core.memcached.12045 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:15 core.memcached.12061 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:15 core.memcached.12075 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:15 core.memcached.12089 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:15 core.memcached.12103 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:16 core.memcached.12117 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:16 core.memcached.12132 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:16 core.memcached.12146 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:16 core.memcached.12160 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:16 core.memcached.12176 rw ------ 1 couchbase couchbase 250097664 Mar 4 13:16 core.memcached.12190 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:17 core.memcached.12204 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:17 core.memcached.12219 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:17 core.memcached.12233 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:17 core.memcached.12247 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:17 core.memcached.12261 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:17 core.memcached.12277 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:18 core.memcached.12291 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:18 core.memcached.12306 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:18 core.memcached.12320 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:18 core.memcached.12502 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:18 core.memcached.12516 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:18 core.memcached.12530 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:19 core.memcached.12544 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:19 core.memcached.12561 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:19 core.memcached.12575 rw ------ 1 couchbase couchbase 250097664 Mar 4 13:19 core.memcached.12843 rw ------ 1 couchbase couchbase 250093568 Mar 4 13:19 core.memcached.13109 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:20 core.memcached.13300 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:21 core.memcached.13490 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:21 core.memcached.13505 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:21 core.memcached.13519 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:21 core.memcached.13533 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:21 core.memcached.13549 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:21 core.memcached.13563 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:22 core.memcached.13577 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:22 core.memcached.13592 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:22 core.memcached.13606 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:22 core.memcached.13620 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:22 core.memcached.13634 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:22 core.memcached.13650 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:23 core.memcached.13664 rw ------ 1 couchbase couchbase 250114048 Mar 4 13:23 core.memcached.13679 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:23 core.memcached.13693 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:23 core.memcached.13707 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:23 core.memcached.13721 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:23 core.memcached.13735 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:24 core.memcached.13749 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:24 core.memcached.13766 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:24 core.memcached.13780 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:24 core.memcached.13946 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:24 core.memcached.13960 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:24 core.memcached.13974 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:25 core.memcached.13988 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:25 core.memcached.14003 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:25 core.memcached.14019 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:25 core.memcached.14033 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:25 core.memcached.14047 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:25 core.memcached.14061 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:26 core.memcached.14147 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:22 core.memcached.1438 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:26 core.memcached.14457 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:22 core.memcached.1452 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:22 core.memcached.1468 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:26 core.memcached.14787 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:23 core.memcached.1482 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:23 core.memcached.1497 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:26 core.memcached.14990 rw ------ 1 couchbase couchbase 250093568 Mar 4 13:26 core.memcached.15006 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:28 core.memcached.15020 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:28 core.memcached.15038 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:29 core.memcached.15052 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:29 core.memcached.15069 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:29 core.memcached.15083 rw ------ 1 couchbase couchbase 250097664 Mar 4 13:29 core.memcached.15097 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:23 core.memcached.1511 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:29 core.memcached.15111 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:29 core.memcached.15125 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:30 core.memcached.15139 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:30 core.memcached.15154 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:30 core.memcached.15170 rw ------ 1 couchbase couchbase 250109952 Mar 4 13:30 core.memcached.15184 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:30 core.memcached.15198 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:30 core.memcached.15212 rw ------ 1 couchbase couchbase 250093568 Mar 4 13:31 core.memcached.15226 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:31 core.memcached.15241 rw ------ 1 couchbase couchbase 250097664 Mar 4 12:23 core.memcached.1525 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:31 core.memcached.15255 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:31 core.memcached.15269 rw ------ 1 couchbase couchbase 250093568 Mar 4 13:31 core.memcached.15285 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:31 core.memcached.15299 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:32 core.memcached.15313 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:32 core.memcached.15328 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:32 core.memcached.15342 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:33 core.memcached.15358 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:33 core.memcached.15373 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:33 core.memcached.15387 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:23 core.memcached.1539 rw ------ 1 couchbase couchbase 250114048 Mar 4 13:33 core.memcached.15411 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:23 core.memcached.1553 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:33 core.memcached.15621 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:24 core.memcached.1567 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:24 core.memcached.1584 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:24 core.memcached.1598 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:33 core.memcached.16007 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:24 core.memcached.1612 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:24 core.memcached.1626 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:34 core.memcached.16327 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:24 core.memcached.1640 rw ------ 1 couchbase couchbase 250114048 Mar 4 13:34 core.memcached.16484 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:34 core.memcached.16498 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:34 core.memcached.16512 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:34 core.memcached.16526 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:25 core.memcached.1654 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:34 core.memcached.16540 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:35 core.memcached.16554 rw ------ 1 couchbase couchbase 101040128 Mar 4 13:35 core.memcached.16569 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:25 core.memcached.1669 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:25 core.memcached.1685 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:25 core.memcached.1699 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:25 core.memcached.1713 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:26 core.memcached.1727 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:26 core.memcached.1742 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:26 core.memcached.1756 rw ------ 1 couchbase couchbase 250097664 Mar 4 12:26 core.memcached.1770 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:26 core.memcached.1786 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:26 core.memcached.1800 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:27 core.memcached.1814 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:27 core.memcached.1829 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:27 core.memcached.1843 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:27 core.memcached.1857 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:27 core.memcached.2076 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:27 core.memcached.2434 rw ------ 1 couchbase couchbase 673476608 Mar 4 11:51 core.memcached.24610 rw ------ 1 couchbase couchbase 252207104 Mar 4 11:53 core.memcached.27971 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:54 core.memcached.27989 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:54 core.memcached.28008 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.28022 rw ------ 1 couchbase couchbase 251142144 Mar 4 11:55 core.memcached.28037 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.28053 rw ------ 1 couchbase couchbase 251158528 Mar 4 11:56 core.memcached.28067 rw ------ 1 couchbase couchbase 251146240 Mar 4 11:57 core.memcached.28084 rw ------ 1 couchbase couchbase 251158528 Mar 4 11:57 core.memcached.28099 rw ------ 1 couchbase couchbase 251158528 Mar 4 11:57 core.memcached.28113 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:58 core.memcached.28127 rw ------ 1 couchbase couchbase 251158528 Mar 4 11:59 core.memcached.28175 rw ------ 1 couchbase couchbase 250097664 Mar 4 12:00 core.memcached.28192 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:00 core.memcached.28207 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:00 core.memcached.28223 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:00 core.memcached.28237 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.28251 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.28279 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:01 core.memcached.28293 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.28307 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.28323 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.28337 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:02 core.memcached.28351 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:02 core.memcached.28366 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:28 core.memcached.2848 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:02 core.memcached.28759 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:02 core.memcached.29221 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:03 core.memcached.29434 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:03 core.memcached.29449 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:03 core.memcached.29463 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:03 core.memcached.29477 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:30 core.memcached.2948 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:03 core.memcached.29491 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:03 core.memcached.29505 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:04 core.memcached.29519 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:04 core.memcached.29536 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:04 core.memcached.29550 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:04 core.memcached.29564 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:04 core.memcached.29578 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:04 core.memcached.29592 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:05 core.memcached.29606 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:05 core.memcached.29621 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:05 core.memcached.29637 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:05 core.memcached.29651 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:05 core.memcached.29665 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:05 core.memcached.29679 rw ------ 1 couchbase couchbase 250097664 Mar 4 12:31 core.memcached.2968 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:06 core.memcached.29693 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:06 core.memcached.29708 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:06 core.memcached.29722 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:06 core.memcached.29736 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:06 core.memcached.29752 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:06 core.memcached.29766 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:07 core.memcached.29780 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:07 core.memcached.29795 rw ------ 1 couchbase couchbase 252211200 Mar 4 12:08 core.memcached.29809 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:32 core.memcached.2983 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:09 core.memcached.30010 rw ------ 1 couchbase couchbase 252211200 Mar 4 12:36 core.memcached.3002 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:09 core.memcached.30027 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:09 core.memcached.30041 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:09 core.memcached.30055 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:10 core.memcached.30069 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:36 core.memcached.3024 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:37 core.memcached.3042 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:10 core.memcached.30466 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:37 core.memcached.3057 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:21 core.memcached.307 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:37 core.memcached.3071 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:37 core.memcached.3085 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:10 core.memcached.30940 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:10 core.memcached.30954 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:10 core.memcached.30968 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:10 core.memcached.30982 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:37 core.memcached.3099 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:11 core.memcached.30996 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:11 core.memcached.31011 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:11 core.memcached.31025 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:11 core.memcached.31039 rw ------ 1 couchbase couchbase 252207104 Mar 4 12:12 core.memcached.31055 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:13 core.memcached.31072 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:13 core.memcached.31087 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:13 core.memcached.31101 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:13 core.memcached.31115 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:13 core.memcached.31129 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:13 core.memcached.31143 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:14 core.memcached.31157 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:37 core.memcached.3116 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:14 core.memcached.31174 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:14 core.memcached.31188 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:14 core.memcached.31202 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:15 core.memcached.31216 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:15 core.memcached.31231 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:15 core.memcached.31247 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:15 core.memcached.31261 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:15 core.memcached.31275 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:15 core.memcached.31289 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:38 core.memcached.3130 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.31303 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:16 core.memcached.31318 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.31332 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.31346 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:38 core.memcached.3145 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:38 core.memcached.3160 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.31683 rw ------ 1 couchbase couchbase 250114048 Mar 4 12:38 core.memcached.3174 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:38 core.memcached.3189 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:16 core.memcached.32045 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:38 core.memcached.3207 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:39 core.memcached.3226 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:21 core.memcached.323 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:17 core.memcached.32414 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:17 core.memcached.32429 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:17 core.memcached.32443 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:17 core.memcached.32457 rw ------ 1 couchbase couchbase 252211200 Mar 4 12:40 core.memcached.3247 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:17 core.memcached.32473 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:18 core.memcached.32487 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:18 core.memcached.32502 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:18 core.memcached.32516 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:18 core.memcached.32530 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:18 core.memcached.32544 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:18 core.memcached.32558 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:19 core.memcached.32572 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:19 core.memcached.32589 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:19 core.memcached.32603 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:19 core.memcached.32617 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:19 core.memcached.32631 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:19 core.memcached.32645 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:20 core.memcached.32659 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:41 core.memcached.3266 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:20 core.memcached.32676 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:20 core.memcached.32690 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:20 core.memcached.32704 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:20 core.memcached.32718 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:21 core.memcached.32732 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:21 core.memcached.32747 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:21 core.memcached.32761 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:42 core.memcached.3281 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:21 core.memcached.343 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:42 core.memcached.3433 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:22 core.memcached.357 rw ------ 1 couchbase couchbase 250114048 Mar 4 12:42 core.memcached.3780 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:42 core.memcached.4028 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:42 core.memcached.4427 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:42 core.memcached.4447 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:43 core.memcached.4461 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:43 core.memcached.4476 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:43 core.memcached.4490 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:43 core.memcached.4504 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:43 core.memcached.4518 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:43 core.memcached.4532 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:44 core.memcached.4546 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:44 core.memcached.4563 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:44 core.memcached.4577 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:44 core.memcached.4591 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:44 core.memcached.4605 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:44 core.memcached.4619 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:45 core.memcached.4633 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:45 core.memcached.4648 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:45 core.memcached.4664 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:45 core.memcached.4678 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:45 core.memcached.4692 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:45 core.memcached.4706 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:46 core.memcached.4720 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:46 core.memcached.4735 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:48 core.memcached.4751 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:48 core.memcached.4769 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:48 core.memcached.4783 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:48 core.memcached.4797 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:49 core.memcached.4811 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:49 core.memcached.4828 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:49 core.memcached.4842 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:49 core.memcached.4856 rw ------ 1 couchbase couchbase 250093568 Mar 4 12:49 core.memcached.4870 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:49 core.memcached.5121 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:50 core.memcached.5386 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:50 core.memcached.5619 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:50 core.memcached.5908 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:50 core.memcached.5967 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:50 core.memcached.5981 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:50 core.memcached.5995 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:51 core.memcached.6009 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:51 core.memcached.6024 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:51 core.memcached.6038 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:51 core.memcached.6052 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:51 core.memcached.6068 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:51 core.memcached.6082 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:52 core.memcached.6096 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:52 core.memcached.6111 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:52 core.memcached.6125 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:52 core.memcached.6139 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:52 core.memcached.6153 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:52 core.memcached.6169 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:53 core.memcached.6183 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:53 core.memcached.6198 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:53 core.memcached.6212 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:53 core.memcached.6226 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:53 core.memcached.6240 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:53 core.memcached.6254 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:54 core.memcached.6268 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:54 core.memcached.6285 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:54 core.memcached.6299 rw ------ 1 couchbase couchbase 250093568 Mar 4 12:54 core.memcached.6313 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:54 core.memcached.6327 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:54 core.memcached.6343 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:55 core.memcached.6357 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:55 core.memcached.6372 rw ------ 1 couchbase couchbase 250114048 Mar 4 12:55 core.memcached.6388 rw ------ 1 couchbase couchbase 250114048 Mar 4 12:55 core.memcached.6585 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:22 core.memcached.668 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:55 core.memcached.6825 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:55 core.memcached.7162 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:56 core.memcached.7366 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:56 core.memcached.7381 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:56 core.memcached.7395 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:56 core.memcached.7409 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:56 core.memcached.7425 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:56 core.memcached.7439 rw ------ 1 couchbase couchbase 250114048 Mar 4 12:57 core.memcached.7453 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:57 core.memcached.7468 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:57 core.memcached.7482 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:57 core.memcached.7496 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:57 core.memcached.7510 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:57 core.memcached.7526 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:58 core.memcached.7540 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:58 core.memcached.7555 rw ------ 1 couchbase couchbase 250114048 Mar 4 12:58 core.memcached.7569 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:58 core.memcached.7583 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:58 core.memcached.7628 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:58 core.memcached.7642 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:59 core.memcached.7656 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:59 core.memcached.7673 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:59 core.memcached.7687 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:59 core.memcached.7701 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:59 core.memcached.7715 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:59 core.memcached.7729 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:00 core.memcached.7743 rw ------ 1 couchbase couchbase 250097664 Mar 4 13:00 core.memcached.7758 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:00 core.memcached.7774 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:00 core.memcached.7788 rw ------ 1 couchbase couchbase 250097664 Mar 4 13:00 core.memcached.7802 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:01 core.memcached.7816 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:01 core.memcached.8196 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:01 core.memcached.8418 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:01 core.memcached.8734 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:01 core.memcached.8912 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:01 core.memcached.8926 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:02 core.memcached.8940 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:02 core.memcached.8955 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:02 core.memcached.8969 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:02 core.memcached.8983 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:02 core.memcached.8997 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:02 core.memcached.9013 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:03 core.memcached.9027 rw ------ 1 couchbase couchbase 250109952 Mar 4 13:03 core.memcached.9042 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:03 core.memcached.9056 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:03 core.memcached.9070 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:03 core.memcached.9084 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:03 core.memcached.9098 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:04 core.memcached.9112 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:04 core.memcached.9129 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:04 core.memcached.9143 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:04 core.memcached.9157 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:04 core.memcached.9171 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:04 core.memcached.9185 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:05 core.memcached.9199 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:05 core.memcached.9214 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:05 core.memcached.9230 rw ------ 1 couchbase couchbase 250109952 Mar 4 13:05 core.memcached.9244 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:05 core.memcached.9258 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:05 core.memcached.9272 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:06 core.memcached.9286 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:06 core.memcached.9301 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:06 core.memcached.9315 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:06 core.memcached.9329 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:06 core.memcached.9464 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:06 core.memcached.9703 rw-r r - 1 jenkins jenkins 126761566 Mar 5 00:40 couchbase-server-enterprise_x86_64_2.0.2-728-rel.rpm drwxrwxrwt 2 couchbase couchbase 4096 Oct 12 06:54 .font-unix srw-rw-rw- 1 couchbase couchbase 0 Oct 12 06:54 .gdm_socket drwxrwxrwt 2 couchbase couchbase 4096 Oct 12 06:54 .ICE-unix drwxrwxrwt 2 couchbase couchbase 4096 Oct 12 06:54 .X11-unix when I removed only a small part of them I got usage 95% This means that after the first сrash, there should not be a problem with disk space [jenkins@cen-1709 tmp] $ sudo rm -rf core.memcached.9* [jenkins@cen-1709 tmp] $ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 55007284 49218324 2949616 95% / /dev/sda1 101086 19983 75884 21% /boot tmpfs 2021844 0 2021844 0% /dev/shm on the second node 10.5.2.14 disk space is OK, but also there are many crashes: [jenkins@cen-1710 tmp] $ ls -la total 27881444 drwxrwxrwt 5 couchbase couchbase 24576 Mar 5 02:04 . drwxr-xr-x 25 root root 4096 Feb 18 03:09 .. rw ------ 1 couchbase couchbase 251162624 Mar 4 12:09 core.memcached.10168 rw ------ 1 couchbase couchbase 250097664 Mar 4 12:09 core.memcached.10220 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:09 core.memcached.10234 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:09 core.memcached.10248 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:09 core.memcached.10262 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:10 core.memcached.10278 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:11 core.memcached.10292 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:11 core.memcached.10307 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:11 core.memcached.10324 rw ------ 1 couchbase couchbase 250093568 Mar 4 12:11 core.memcached.10338 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:12 core.memcached.10352 rw ------ 1 couchbase couchbase 250097664 Mar 4 12:12 core.memcached.10366 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:12 core.memcached.10381 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:12 core.memcached.10397 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:12 core.memcached.10411 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:13 core.memcached.10425 rw ------ 1 couchbase couchbase 250093568 Mar 4 12:14 core.memcached.10442 rw ------ 1 couchbase couchbase 250093568 Mar 4 12:14 core.memcached.10456 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:14 core.memcached.10471 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:14 core.memcached.10485 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:14 core.memcached.10499 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:15 core.memcached.10513 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.10530 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.10544 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:16 core.memcached.10561 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:16 core.memcached.10575 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:16 core.memcached.10589 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:16 core.memcached.10666 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:17 core.memcached.11045 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:17 core.memcached.11388 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:17 core.memcached.11670 rw ------ 1 couchbase couchbase 250093568 Mar 4 12:17 core.memcached.11686 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:19 core.memcached.11700 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:20 core.memcached.11749 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:20 core.memcached.11763 rw ------ 1 couchbase couchbase 250093568 Mar 4 12:20 core.memcached.11778 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:20 core.memcached.11792 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:20 core.memcached.11806 rw ------ 1 couchbase couchbase 252207104 Mar 4 12:23 core.memcached.11820 rw ------ 1 couchbase couchbase 252207104 Mar 4 12:25 core.memcached.11843 rw ------ 1 couchbase couchbase 252207104 Mar 4 12:28 core.memcached.11861 rw ------ 1 couchbase couchbase 250093568 Mar 4 12:28 core.memcached.11882 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:28 core.memcached.11896 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:28 core.memcached.11910 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:28 core.memcached.11926 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:29 core.memcached.11940 rw ------ 1 couchbase couchbase 250097664 Mar 4 12:29 core.memcached.11954 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:29 core.memcached.11969 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:29 core.memcached.11983 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:29 core.memcached.11997 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:29 core.memcached.12011 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:30 core.memcached.12027 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:30 core.memcached.12041 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:30 core.memcached.12056 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:30 core.memcached.12070 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:30 core.memcached.12084 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:31 core.memcached.12116 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:31 core.memcached.12779 rw ------ 1 couchbase couchbase 250114048 Mar 4 12:31 core.memcached.13085 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:31 core.memcached.13167 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:31 core.memcached.13181 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:31 core.memcached.13195 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:32 core.memcached.13209 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:32 core.memcached.13223 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:32 core.memcached.13238 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:32 core.memcached.13254 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:32 core.memcached.13268 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:32 core.memcached.13282 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:33 core.memcached.13296 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:33 core.memcached.13310 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:33 core.memcached.13325 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:33 core.memcached.13339 rw ------ 1 couchbase couchbase 250089472 Mar 4 12:33 core.memcached.13353 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:33 core.memcached.13369 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:34 core.memcached.13383 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:34 core.memcached.13397 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:34 core.memcached.13412 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:34 core.memcached.13571 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:34 core.memcached.13590 rw ------ 1 couchbase couchbase 250093568 Mar 4 12:34 core.memcached.13604 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:35 core.memcached.13620 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:35 core.memcached.13634 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:35 core.memcached.13649 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:35 core.memcached.13663 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:35 core.memcached.13677 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:35 core.memcached.13691 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:36 core.memcached.13705 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:36 core.memcached.13719 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:36 core.memcached.13736 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:36 core.memcached.13983 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:36 core.memcached.14322 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:39 core.memcached.14652 rw ------ 1 couchbase couchbase 250093568 Mar 4 12:39 core.memcached.14672 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:39 core.memcached.14687 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:39 core.memcached.14701 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:39 core.memcached.14715 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:39 core.memcached.14729 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:40 core.memcached.14745 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:40 core.memcached.14759 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:40 core.memcached.14774 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:40 core.memcached.14788 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:40 core.memcached.14802 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:40 core.memcached.14816 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:41 core.memcached.14980 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:41 core.memcached.14994 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:41 core.memcached.15011 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:41 core.memcached.15025 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:41 core.memcached.15039 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:41 core.memcached.15053 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:42 core.memcached.15067 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:42 core.memcached.15081 rw ------ 1 couchbase couchbase 252194816 Mar 4 12:46 core.memcached.15096 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:46 core.memcached.15122 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:46 core.memcached.15136 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:46 core.memcached.15150 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:46 core.memcached.15164 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:47 core.memcached.15178 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:47 core.memcached.15192 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:47 core.memcached.15207 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:47 core.memcached.15223 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:47 core.memcached.15237 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:47 core.memcached.15251 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:48 core.memcached.15383 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:48 core.memcached.15781 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:48 core.memcached.16246 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:49 core.memcached.16332 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:49 core.memcached.16348 rw ------ 1 couchbase couchbase 252211200 Mar 4 12:51 core.memcached.16363 rw ------ 1 couchbase couchbase 252194816 Mar 4 12:53 core.memcached.16383 rw ------ 1 couchbase couchbase 252207104 Mar 4 12:59 core.memcached.16403 rw ------ 1 couchbase couchbase 252207104 Mar 4 13:00 core.memcached.16431 rw ------ 1 couchbase couchbase 250109952 Mar 4 13:01 core.memcached.16448 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:01 core.memcached.16475 rw ------ 1 couchbase couchbase 252207104 Mar 4 13:05 core.memcached.16492 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:05 core.memcached.16518 rw ------ 1 couchbase couchbase 250109952 Mar 4 13:05 core.memcached.16564 rw ------ 1 couchbase couchbase 252207104 Mar 4 13:07 core.memcached.16578 rw ------ 1 couchbase couchbase 250093568 Mar 4 13:08 core.memcached.16598 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:08 core.memcached.16612 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:08 core.memcached.16627 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:08 core.memcached.16641 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:08 core.memcached.16655 rw ------ 1 couchbase couchbase 250114048 Mar 4 13:08 core.memcached.16671 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:09 core.memcached.16685 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:10 core.memcached.16699 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:10 core.memcached.17164 rw ------ 1 couchbase couchbase 252190720 Mar 4 13:19 core.memcached.17475 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:20 core.memcached.17794 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:20 core.memcached.17808 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:20 core.memcached.17822 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:20 core.memcached.17837 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:20 core.memcached.17851 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:20 core.memcached.17865 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:21 core.memcached.17879 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:21 core.memcached.17893 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:21 core.memcached.17909 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:21 core.memcached.17924 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:21 core.memcached.17938 rw ------ 1 couchbase couchbase 252207104 Mar 4 13:24 core.memcached.17952 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:25 core.memcached.17972 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:26 core.memcached.17990 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:26 core.memcached.18007 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:27 core.memcached.18021 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:27 core.memcached.18188 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:27 core.memcached.18202 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:27 core.memcached.18219 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:27 core.memcached.18233 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:27 core.memcached.18247 rw ------ 1 couchbase couchbase 250114048 Mar 4 13:28 core.memcached.18261 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:28 core.memcached.18275 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:28 core.memcached.18289 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:28 core.memcached.18304 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:28 core.memcached.18649 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:28 core.memcached.18980 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:29 core.memcached.19219 rw ------ 1 couchbase couchbase 251162624 Mar 4 13:30 core.memcached.19233 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:30 core.memcached.19250 rw ------ 1 couchbase couchbase 251146240 Mar 4 13:31 core.memcached.19265 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:32 core.memcached.19282 rw ------ 1 couchbase couchbase 250114048 Mar 4 13:32 core.memcached.19299 rw ------ 1 couchbase couchbase 250114048 Mar 4 13:32 core.memcached.19313 rw ------ 1 couchbase couchbase 252207104 Mar 4 13:34 core.memcached.19327 rw ------ 1 couchbase couchbase 251158528 Mar 4 13:35 core.memcached.19497 rw ------ 1 couchbase couchbase 251142144 Mar 4 13:35 core.memcached.19513 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:44 core.memcached.6300 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:45 core.memcached.6328 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:45 core.memcached.6342 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:45 core.memcached.6357 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:45 core.memcached.6371 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:45 core.memcached.6385 rw ------ 1 couchbase couchbase 250097664 Mar 4 11:45 core.memcached.6399 rw ------ 1 couchbase couchbase 250097664 Mar 4 11:46 core.memcached.6413 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:46 core.memcached.6427 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:46 core.memcached.6444 rw ------ 1 couchbase couchbase 250097664 Mar 4 11:46 core.memcached.6458 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:46 core.memcached.6472 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:46 core.memcached.6486 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:47 core.memcached.6500 rw ------ 1 couchbase couchbase 250093568 Mar 4 11:47 core.memcached.6514 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:47 core.memcached.6529 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:47 core.memcached.6545 rw ------ 1 couchbase couchbase 250097664 Mar 4 11:47 core.memcached.6559 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:47 core.memcached.6573 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:48 core.memcached.6587 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:48 core.memcached.6601 rw ------ 1 couchbase couchbase 250097664 Mar 4 11:48 core.memcached.6616 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:48 core.memcached.6630 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:48 core.memcached.6644 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:48 core.memcached.6686 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:49 core.memcached.6700 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:49 core.memcached.6714 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:49 core.memcached.6729 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:49 core.memcached.6743 rw ------ 1 couchbase couchbase 250093568 Mar 4 11:49 core.memcached.6757 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:49 core.memcached.6771 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:50 core.memcached.6787 rw ------ 1 couchbase couchbase 250093568 Mar 4 11:50 core.memcached.6801 rw ------ 1 couchbase couchbase 250093568 Mar 4 11:50 core.memcached.6816 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:50 core.memcached.6830 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:50 core.memcached.6844 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:50 core.memcached.6858 rw ------ 1 couchbase couchbase 250097664 Mar 4 11:51 core.memcached.6872 rw ------ 1 couchbase couchbase 250093568 Mar 4 11:51 core.memcached.6886 rw ------ 1 couchbase couchbase 250093568 Mar 4 11:51 core.memcached.6903 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:51 core.memcached.6917 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:51 core.memcached.6931 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:51 core.memcached.6945 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:52 core.memcached.6959 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:52 core.memcached.6973 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:52 core.memcached.6988 rw ------ 1 couchbase couchbase 250097664 Mar 4 11:52 core.memcached.7004 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:52 core.memcached.7018 rw ------ 1 couchbase couchbase 250093568 Mar 4 11:52 core.memcached.7032 rw ------ 1 couchbase couchbase 250093568 Mar 4 11:53 core.memcached.7046 rw ------ 1 couchbase couchbase 250097664 Mar 4 11:53 core.memcached.7060 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:53 core.memcached.7075 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:53 core.memcached.7089 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:53 core.memcached.7103 rw ------ 1 couchbase couchbase 250097664 Mar 4 11:53 core.memcached.7119 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:54 core.memcached.7133 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:54 core.memcached.7147 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:54 core.memcached.7162 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:54 core.memcached.7176 rw ------ 1 couchbase couchbase 250114048 Mar 4 11:54 core.memcached.7190 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.7204 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.7220 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.7235 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.7249 rw ------ 1 couchbase couchbase 251142144 Mar 4 11:55 core.memcached.7263 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:55 core.memcached.7277 rw ------ 1 couchbase couchbase 251146240 Mar 4 11:56 core.memcached.7291 rw ------ 1 couchbase couchbase 251158528 Mar 4 11:56 core.memcached.7305 rw ------ 1 couchbase couchbase 251158528 Mar 4 11:56 core.memcached.7322 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:56 core.memcached.7336 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:56 core.memcached.7350 rw ------ 1 couchbase couchbase 251158528 Mar 4 11:57 core.memcached.7364 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:57 core.memcached.7378 rw ------ 1 couchbase couchbase 251158528 Mar 4 11:57 core.memcached.7393 rw ------ 1 couchbase couchbase 251158528 Mar 4 11:57 core.memcached.7409 rw ------ 1 couchbase couchbase 251146240 Mar 4 11:57 core.memcached.7423 rw ------ 1 couchbase couchbase 251158528 Mar 4 11:57 core.memcached.7437 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:58 core.memcached.7451 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:58 core.memcached.7465 rw ------ 1 couchbase couchbase 251158528 Mar 4 11:58 core.memcached.7565 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:58 core.memcached.7701 rw ------ 1 couchbase couchbase 251158528 Mar 4 11:58 core.memcached.7715 rw ------ 1 couchbase couchbase 251142144 Mar 4 11:59 core.memcached.7731 rw ------ 1 couchbase couchbase 250109952 Mar 4 11:59 core.memcached.7745 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:59 core.memcached.7760 rw ------ 1 couchbase couchbase 251146240 Mar 4 11:59 core.memcached.7774 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:59 core.memcached.7788 rw ------ 1 couchbase couchbase 251162624 Mar 4 11:59 core.memcached.7802 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:00 core.memcached.7818 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:00 core.memcached.7832 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:00 core.memcached.7947 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:00 core.memcached.8405 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:00 core.memcached.8692 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:01 core.memcached.8706 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.8736 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:01 core.memcached.8750 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:01 core.memcached.8766 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:02 core.memcached.8780 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:02 core.memcached.8794 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:02 core.memcached.8809 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:02 core.memcached.8825 rw ------ 1 couchbase couchbase 252207104 Mar 4 12:04 core.memcached.8839 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:05 core.memcached.8857 rw ------ 1 couchbase couchbase 250109952 Mar 4 12:06 core.memcached.8906 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:06 core.memcached.8920 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:06 core.memcached.8937 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:06 core.memcached.8951 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:06 core.memcached.8965 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:06 core.memcached.8979 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:07 core.memcached.8993 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:07 core.memcached.9007 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:07 core.memcached.9022 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:07 core.memcached.9038 rw ------ 1 couchbase couchbase 250093568 Mar 4 12:07 core.memcached.9052 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:07 core.memcached.9066 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:08 core.memcached.9080 rw ------ 1 couchbase couchbase 250114048 Mar 4 12:08 core.memcached.9094 rw ------ 1 couchbase couchbase 251162624 Mar 4 12:08 core.memcached.9109 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:08 core.memcached.9123 rw ------ 1 couchbase couchbase 251146240 Mar 4 12:08 core.memcached.9137 rw ------ 1 couchbase couchbase 251158528 Mar 4 12:08 core.memcached.9262 rw ------ 1 couchbase couchbase 251142144 Mar 4 12:09 core.memcached.9740 rw-r r - 1 jenkins jenkins 126761566 Mar 5 00:40 couchbase-server-enterprise_x86_64_2.0.2-728-rel.rpm drwxrwxrwt 2 couchbase couchbase 4096 Oct 11 08:01 .font-unix srw-rw-rw- 1 couchbase couchbase 0 Oct 11 08:01 .gdm_socket drwxrwxrwt 2 couchbase couchbase 4096 Oct 11 08:01 .ICE-unix drwxrwxrwt 2 couchbase couchbase 4096 Oct 11 08:01 .X11-unix [jenkins@cen-1710 tmp] $ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 55007284 36258700 15909240 70% / /dev/sda1 101086 19983 75884 21% /boot tmpfs 2021844 0 2021844 0% /dev/shm [jenkins@cen-1709 tmp] $ sudo rm -rf core.memcached.9* [jenkins@cen-1709 tmp] $ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 55007284 49218324 2949616 95% / /dev/sda1 101086 19983 75884 21% /boot tmpfs 2021844 0 2021844 0% /dev/shm [jenkins@cen-1709 tmp] $ sudo gdb /opt/couchbase/bin/memcached core.memcached.24610 GNU gdb (GDB) CentOS (7.0.1-45.el5.centos) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: < http://www.gnu.org/software/gdb/bugs/ >... Reading symbols from /opt/couchbase/bin/memcached...done. warning: exec file is newer than core file. [New Thread 24624] [New Thread 27925] [New Thread 27924] [New Thread 27923] [New Thread 27922] [New Thread 27921] [New Thread 24626] [New Thread 24625] [New Thread 24623] [New Thread 24622] [New Thread 24621] [New Thread 24619] [New Thread 24618] [New Thread 24610] Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done. Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done. Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so Reading symbols from /opt/couchbase/lib/memcached/ep.so...done. Loaded symbols for /opt/couchbase/lib/memcached/ep.so Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done. Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1 Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done. Loaded symbols for /opt/couchbase/lib/libsnappy.so.1 Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done. Loaded symbols for /opt/couchbase/lib/libicuuc.so.44 Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/libicudata.so.44 Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done. Loaded symbols for /opt/couchbase/lib/libicui18n.so.44 Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff39fac000 Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'. Program terminated with signal 11, Segmentation fault. #0 0x00002aaaaaf0eb1a in LOG (severity=<value optimized out>, fmt=<value optimized out>) at src/ep_engine.cc:1184 1184 src/ep_engine.cc: No such file or directory. in src/ep_engine.cc (gdb) t a a bt Thread 14 (Thread 0x2baa8c1f7220 (LWP 24610)): #0 0x0000003f142d48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002baa8bd74576 in epoll_dispatch (base=0xc916000, tv=<value optimized out>) at epoll.c:404 #2 0x00002baa8bd5fe44 in event_base_loop (base=0xc916000, flags=<value optimized out>) at event.c:1558 #3 0x0000000000409742 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7918 Thread 13 (Thread 24618): #0 0x0000003f142c678b in read () from /lib64/libc.so.6 #1 0x0000003f1426cd57 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x0000003f1426d71e in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x0000003f14262804 in _IO_getline_info_internal () from /lib64/libc.so.6 #4 0x0000003f142616a9 in fgets () from /lib64/libc.so.6 #5 0x00002baa8c1f8939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37 #6 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0 #7 0x0000003f142d44bd in clone () from /lib64/libc.so.6 Thread 12 (Thread 24619): #0 0x0000003f14e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaaaae4d6 in logger_thead_main (arg=0x80b2040) at extensions/loggers/file_logger.c:368 #2 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0 #3 0x0000003f142d44bd in clone () from /lib64/libc.so.6 Thread 11 (Thread 24621): #0 0x0000003f142d48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002baa8bd74576 in epoll_dispatch (base=0xc916500, tv=<value optimized out>) at epoll.c:404 #2 0x00002baa8bd5fe44 in event_base_loop (base=0xc916500, flags=<value optimized out>) at event.c:1558 #3 0x0000000000414504 in worker_libevent (arg=0x80b5900) at daemon/thread.c:301 #4 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f142d44bd in clone () from /lib64/libc.so.6 Thread 10 (Thread 24622): #0 0x0000003f142d48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002baa8bd74576 in epoll_dispatch (base=0xc916280, tv=<value optimized out>) at epoll.c:404 #2 0x00002baa8bd5fe44 in event_base_loop (base=0xc916280, flags=<value optimized out>) at event.c:1558 #3 0x0000000000414504 in worker_libevent (arg=0x80b59f8) at daemon/thread.c:301 #4 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f142d44bd in clone () from /lib64/libc.so.6 Thread 9 (Thread 24623): #0 0x0000003f142d48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002baa8bd74576 in epoll_dispatch (base=0xc916c80, tv=<value optimized out>) at epoll.c:404 #2 0x00002baa8bd5fe44 in event_base_loop (base=0xc916c80, flags=<value optimized out>) at event.c:1558 #3 0x0000000000414504 in worker_libevent (arg=0x80b5af0) at daemon/thread.c:301 #4 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f142d44bd in clone () from /lib64/libc.so.6 Thread 8 (Thread 24625): #0 0x0000003f142d48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002baa8bd74576 in epoll_dispatch (base=0xc916780, tv=<value optimized out>) at epoll.c:404 #2 0x00002baa8bd5fe44 in event_base_loop (base=0xc916780, flags=<value optimized out>) at event.c:1558 #3 0x0000000000414504 in worker_libevent (arg=0x80b5ce0) at daemon/thread.c:301 #4 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f142d44bd in clone () from /lib64/libc.so.6 -- Type <return> to continue, or q <return> to quit -- Thread 7 (Thread 24626): #0 0x0000003f1429a541 in nanosleep () from /lib64/libc.so.6 #1 0x0000003f142cded4 in usleep () from /lib64/libc.so.6 #2 0x00002aaaaaf30255 in updateStatsThread (arg=0x80b24c0) at src/memory_tracker.cc:31 #3 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f142d44bd in clone () from /lib64/libc.so.6 Thread 6 (Thread 27921): #0 0x0000003f14e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaaef2e58 in wait (this=0xc963440, d=...) at src/syncobject.hh:57 #2 IdleTask::run (this=0xc963440, d=...) at src/dispatcher.cc:328 #3 0x00002aaaaaef568a in Dispatcher::run (this=0x154d6a80) at src/dispatcher.cc:171 #4 0x00002aaaaaef5f3d in launch_dispatcher_thread (arg=0x154d6a80) at src/dispatcher.cc:28 #5 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0 #6 0x0000003f142d44bd in clone () from /lib64/libc.so.6 Thread 5 (Thread 27922): #0 0x0000003f14e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaaef2e58 in wait (this=0xc9637a0, d=...) at src/syncobject.hh:57 #2 IdleTask::run (this=0xc9637a0, d=...) at src/dispatcher.cc:328 #3 0x00002aaaaaef568a in Dispatcher::run (this=0x154d68c0) at src/dispatcher.cc:171 #4 0x00002aaaaaef5f3d in launch_dispatcher_thread (arg=0x154d68c0) at src/dispatcher.cc:28 #5 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0 #6 0x0000003f142d44bd in clone () from /lib64/libc.so.6 Thread 4 (Thread 27923): #0 0x0000003f14e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaaef2e58 in wait (this=0xc962c60, d=...) at src/syncobject.hh:57 #2 IdleTask::run (this=0xc962c60, d=...) at src/dispatcher.cc:328 #3 0x00002aaaaaef568a in Dispatcher::run (this=0x154d6700) at src/dispatcher.cc:171 #4 0x00002aaaaaef5f3d in launch_dispatcher_thread (arg=0x154d6700) at src/dispatcher.cc:28 #5 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0 #6 0x0000003f142d44bd in clone () from /lib64/libc.so.6 Thread 3 (Thread 27924): #0 0x0000003f14e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaaef2e58 in wait (this=0xc9633b0, d=...) at src/syncobject.hh:57 #2 IdleTask::run (this=0xc9633b0, d=...) at src/dispatcher.cc:328 #3 0x00002aaaaaef568a in Dispatcher::run (this=0x154d6540) at src/dispatcher.cc:171 #4 0x00002aaaaaef5f3d in launch_dispatcher_thread (arg=0x154d6540) at src/dispatcher.cc:28 #5 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0 #6 0x0000003f142d44bd in clone () from /lib64/libc.so.6 Thread 2 (Thread 27925): #0 0x0000003f14e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaaf1028f in wait (this=0x14550900) at src/syncobject.hh:57 #2 wait (this=0x14550900) at src/syncobject.hh:73 #3 wait (this=0x14550900) at src/tapconnmap.hh:169 #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x14550900) at src/ep_engine.cc:3406 #5 0x00002aaaaaf10373 in EvpNotifyPendingConns (arg=0x14550900) at src/ep_engine.cc:1139 #6 0x0000003f14e0673d in start_thread () from /lib64/libpthread.so.0 #7 0x0000003f142d44bd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x4477a940 (LWP 24624)): #0 0x00002aaaaaf0eb1a in LOG (severity=<value optimized out>, fmt=<value optimized out>) at src/ep_engine.cc:1184 -- Type <return> to continue, or q <return> to quit -- #1 0x34202c343634202c in ?? () #2 0x2c363634202c3536 in ?? () #3 0x3634202c37363420 in ?? () #4 0x202c393634202c38 in ?? () #5 0x313734202c303734 in ?? () #6 0x34202c323734202c in ?? () #7 0x2c343734202c3337 in ?? () #8 0x3734202c35373420 in ?? () #9 0x202c373734202c36 in ?? () #10 0x393734202c383734 in ?? () #11 0x34202c303834202c in ?? () #12 0x2c323834202c3138 in ?? () #13 0x3834202c33383420 in ?? () #14 0x202c353834202c34 in ?? () #15 0x373834202c363834 in ?? () #16 0x34202c383834202c in ?? () #17 0x2c303934202c3938 in ?? () #18 0x3934202c31393420 in ?? () #19 0x202c333934202c32 in ?? () #20 0x353934202c343934 in ?? () #21 0x34202c363934202c in ?? () #22 0x2c383934202c3739 in ?? () #23 0x3035202c39393420 in ?? () #24 0x202c313035202c30 in ?? () #25 0x333035202c323035 in ?? () #26 0x35202c343035202c in ?? () #27 0x2c363035202c3530 in ?? () #28 0x3035202c37303520 in ?? () #29 0x202c393035202c38 in ?? () #30 0x313135202c303135 in ?? () #31 0x00000000000a202c in ?? () #32 0x0000000000000000 in ?? () don't have collect info due to out of space on slave : IMPOSSIBLE TO GRAB CBCOLLECT FROM 10.1.3.118 Traceback (most recent call last): File "testrunner", line 362, in <module> xunit.write(" {0} /report- {1} ".format(logs_folder, str_time)) File "lib/xunit.py", line 102, in write report_xml_file.close() IOError: [Errno 28] No space left on device core dumps: jenkins(QA keys)@10.5.2.14:/tmp/backup/ MB-7652 _core.memcached.6300 jenkins(QA keys)@10.5.2.13:/tmp/backup/ MB-7652 _core.memcached.24610 Mike, whether the lack of disk space causes crashes?
        Hide
        andreibaranouski Andrei Baranouski added a comment -

        build 2.0.2-727-rel

        Show
        andreibaranouski Andrei Baranouski added a comment - build 2.0.2-727-rel
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ep-engine-2-0 #481 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/481/)
        MB-7652: Check for buffer overflow when logging messages (Revision 23d2e8d6535e48305ba629f42381b2fb5317a641)

        Result = SUCCESS
        Mike Wiederhold :
        Files :

        • src/ep_engine.cc
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #481 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/481/ ) MB-7652 : Check for buffer overflow when logging messages (Revision 23d2e8d6535e48305ba629f42381b2fb5317a641) Result = SUCCESS Mike Wiederhold : Files : src/ep_engine.cc
        Hide
        maria Maria McDuff (Inactive) added a comment -

        pls verify in current 2.0.2 build.

        Show
        maria Maria McDuff (Inactive) added a comment - pls verify in current 2.0.2 build.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        ok to close. fixed by mike. QE hasn't seen for at least a week.

        Show
        maria Maria McDuff (Inactive) added a comment - ok to close. fixed by mike. QE hasn't seen for at least a week.

          People

          • Assignee:
            Chisheng Chisheng Hong (Inactive)
            Reporter:
            andreibaranouski Andrei Baranouski
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes