Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-18766

Core files created by go-processes for every crash fill up disk

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 4.6.0
    • 4.5.0
    • build
    • None
    • Untriaged
    • Unknown

    Description

      Build
      4.5.0-1818

      Pls see below at the list of huge core files generated -
      total 39638308
      rw------ 1 couchbase couchbase 811401216 Mar 10 18:11 core.cbft.23875
      rw------ 1 couchbase couchbase 724312064 Mar 10 18:11 core.cbft.24082
      rw------ 1 couchbase couchbase 649412608 Mar 10 18:11 core.cbft.24117
      rw------ 1 couchbase couchbase 723832832 Mar 10 18:12 core.cbft.24235
      rw------ 1 couchbase couchbase 722911232 Mar 10 18:12 core.cbft.24267
      rw------ 1 couchbase couchbase 662159360 Mar 10 18:12 core.cbft.24301
      rw------ 1 couchbase couchbase 679763968 Mar 10 18:12 core.cbft.24370
      rw------ 1 couchbase couchbase 711233536 Mar 10 18:12 core.cbft.24404
      rw------ 1 couchbase couchbase 696569856 Mar 10 18:12 core.cbft.24438
      rw------ 1 couchbase couchbase 698372096 Mar 10 18:12 core.cbft.24475
      rw------ 1 couchbase couchbase 723865600 Mar 10 18:13 core.cbft.24510
      rw------ 1 couchbase couchbase 744476672 Mar 10 18:13 core.cbft.24546
      rw------ 1 couchbase couchbase 689987584 Mar 10 18:13 core.cbft.24583
      rw------ 1 couchbase couchbase 673005568 Mar 10 18:13 core.cbft.24666
      rw------ 1 couchbase couchbase 673943552 Mar 10 18:13 core.cbft.24697
      rw------ 1 couchbase couchbase 693260288 Mar 10 18:13 core.cbft.24731
      rw------ 1 couchbase couchbase 698220544 Mar 10 18:13 core.cbft.24764
      rw------ 1 couchbase couchbase 690896896 Mar 10 18:14 core.cbft.24798
      rw------ 1 couchbase couchbase 686518272 Mar 10 18:14 core.cbft.24831
      rw------ 1 couchbase couchbase 660963328 Mar 10 18:14 core.cbft.24919
      rw------ 1 couchbase couchbase 668897280 Mar 10 18:14 core.cbft.25040
      rw------ 1 couchbase couchbase 675119104 Mar 10 18:14 core.cbft.25101
      rw------ 1 couchbase couchbase 661164032 Mar 10 18:14 core.cbft.25159
      rw------ 1 couchbase couchbase 683016192 Mar 10 18:14 core.cbft.25388
      rw------ 1 couchbase couchbase 680570880 Mar 10 18:15 core.cbft.25553
      rw------ 1 couchbase couchbase 683446272 Mar 10 18:15 core.cbft.25759
      rw------ 1 couchbase couchbase 651444224 Mar 10 18:15 core.cbft.25985
      rw------ 1 couchbase couchbase 662052864 Mar 10 18:15 core.cbft.26308
      rw------ 1 couchbase couchbase 670261248 Mar 10 18:15 core.cbft.26378
      rw------ 1 couchbase couchbase 693633024 Mar 10 18:15 core.cbft.26411
      rw------ 1 couchbase couchbase 676925440 Mar 10 18:16 core.cbft.26445
      rw------ 1 couchbase couchbase 664879104 Mar 10 18:16 core.cbft.26479
      rw------ 1 couchbase couchbase 662908928 Mar 10 18:16 core.cbft.26511
      rw------ 1 couchbase couchbase 643194880 Mar 10 18:16 core.cbft.26579
      rw------ 1 couchbase couchbase 676065280 Mar 10 18:16 core.cbft.26610
      rw------ 1 couchbase couchbase 687312896 Mar 10 18:16 core.cbft.26642
      rw------ 1 couchbase couchbase 677015552 Mar 10 18:16 core.cbft.26674
      rw------ 1 couchbase couchbase 689393664 Mar 10 18:17 core.cbft.26710
      rw------ 1 couchbase couchbase 703168512 Mar 10 18:17 core.cbft.26744
      rw------ 1 couchbase couchbase 680656896 Mar 10 18:17 core.cbft.26777
      rw------ 1 couchbase couchbase 677236736 Mar 10 18:17 core.cbft.26859
      rw------ 1 couchbase couchbase 671997952 Mar 10 18:17 core.cbft.26891
      rw------ 1 couchbase couchbase 676569088 Mar 10 18:17 core.cbft.26922
      rw------ 1 couchbase couchbase 699981824 Mar 10 18:18 core.cbft.26955
      rw------ 1 couchbase couchbase 677490688 Mar 10 18:18 core.cbft.26990
      rw------ 1 couchbase couchbase 663830528 Mar 10 18:18 core.cbft.27023
      rw------ 1 couchbase couchbase 684744704 Mar 10 18:18 core.cbft.27091
      rw------ 1 couchbase couchbase 664371200 Mar 10 18:18 core.cbft.27124
      rw------ 1 couchbase couchbase 649760768 Mar 10 18:18 core.cbft.27157
      rw------ 1 couchbase couchbase 669753344 Mar 10 18:18 core.cbft.27188
      rw------ 1 couchbase couchbase 664756224 Mar 10 18:19 core.cbft.27221
      rw------ 1 couchbase couchbase 666906624 Mar 10 18:19 core.cbft.27254
      rw------ 1 couchbase couchbase 675340288 Mar 10 18:19 core.cbft.27285
      rw------ 1 couchbase couchbase 668155904 Mar 10 18:19 core.cbft.27367
      rw------ 1 couchbase couchbase 675057664 Mar 10 18:19 core.cbft.27398
      rw------ 1 couchbase couchbase 691658752 Mar 10 18:19 core.cbft.27432
      rw------ 1 couchbase couchbase 654942208 Mar 10 18:19 core.cbft.27465
      rw------ 1 couchbase couchbase 667906048 Mar 10 18:20 core.cbft.27498
      rw------ 1 couchbase couchbase 677924864 Mar 10 18:20 core.cbft.27530
      rw------ 1 couchbase couchbase 913321984 Mar 10 18:34 core.cbft.29599
      rw------ 1 couchbase couchbase 1381285888 Mar 11 14:55 core.cbft.1318
      rw------ 1 couchbase couchbase 1263468544 Mar 11 15:10 core.cbft.4907
      rw------ 1 couchbase couchbase 1325338624 Mar 11 15:10 core.cbft.5042
      rw------ 1 couchbase couchbase 1250242560 Mar 11 15:10 core.cbft.5125
      rw------ 1 couchbase couchbase 1000153088 Mar 11 15:10 core.cbft.5197
      rw------ 1 couchbase couchbase 1087172608 Mar 11 15:11 core.cbft.5252
      rw------ 1 couchbase couchbase 976760832 Mar 11 15:11 core.cbft.5349
      rw------ 1 couchbase couchbase 905543680 Mar 11 15:11 core.cbft.5476
      rw------ 1 couchbase couchbase 952102912 Mar 11 15:11 core.cbft.5538
      rw------ 1 couchbase couchbase 723116032 Mar 11 15:11 core.cbft.5590
      rw------ 1 couchbase couchbase 881520640 Mar 11 15:11 core.cbft.5626
      rw------ 1 couchbase couchbase 945836032 Mar 11 15:11 core.cbft.5676
      rw------ 1 couchbase couchbase 946434048 Mar 11 15:11 core.cbft.5726
      rw------ 1 couchbase couchbase 947052544 Mar 11 15:12 core.cbft.5777
      rw------ 1 couchbase couchbase 753352704 Mar 11 15:12 core.cbft.5878
      rw------ 1 couchbase couchbase 913555456 Mar 11 15:12 core.cbft.5916
      rw------ 1 couchbase couchbase 740196352 Mar 11 15:12 core.cbft.5964
      rw------ 1 couchbase couchbase 883363840 Mar 11 15:12 core.cbft.6001
      rw------ 1 couchbase couchbase 878178304 Mar 11 15:12 core.cbft.6045
      rw------ 1 couchbase couchbase 817782784 Mar 11 15:12 core.cbft.6092
      rw------ 1 couchbase couchbase 749252608 Mar 11 15:12 core.cbft.6135
      rw------ 1 couchbase couchbase 975790080 Mar 11 15:13 core.cbft.6177
      rw------ 1 couchbase couchbase 781348864 Mar 11 15:13 core.cbft.6264
      rw------ 1 couchbase couchbase 959569920 Mar 11 15:13 core.cbft.6305
      rw------ 1 couchbase couchbase 989573120 Mar 11 15:13 core.cbft.6357
      rw------ 1 couchbase couchbase 796274688 Mar 11 15:13 core.cbft.6410
      rw------ 1 couchbase couchbase 860684288 Mar 11 15:13 core.cbft.6452
      rw------ 1 couchbase couchbase 891793408 Mar 11 15:13 core.cbft.6493
      rw------ 1 couchbase couchbase 891367424 Mar 11 15:13 core.cbft.6541
      rw------ 1 couchbase couchbase 884666368 Mar 11 15:14 core.cbft.6588
      rw------ 1 couchbase couchbase 900722688 Mar 11 15:14 core.cbft.6634
      rw------ 1 couchbase couchbase 850415616 Mar 11 15:14 core.cbft.6781
      rw------ 1 couchbase couchbase 837189632 Mar 11 15:14 core.cbft.6842
      rw------ 1 couchbase couchbase 827715584 Mar 11 15:14 core.cbft.6958
      rw------ 1 couchbase couchbase 716410880 Mar 11 15:14 core.cbft.7003
      rw------ 1 couchbase couchbase 881455104 Mar 11 15:14 core.cbft.7066
      rw------ 1 couchbase couchbase 39526400 Mar 11 15:14 core.cbft.7313
      rw------ 1 couchbase couchbase 0 Mar 11 15:16 core.cbft.9520

      Ofcourse, the underlying problem causing the crash must be fixed, but there are going to be some crashes that are only visible in the field. Multiple crashes filling up disk space could be a serious problem.

      I understand this was turned off after discussion in Sherlock but was turned on again in Watson to help with forestdb diagnosis but it turns out the core dumps are still missing cgo info and aren't completely useful either.

      In test environments, currently this is a big problem with all tests failing after 'n' crashes.

      Creating this MB as a placeholder to turn off GOTRACEBACK=crash after discussion with teams.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              apiravi Aruna Piravi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty