Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46576

[System Test]: Seeing memory leak in one of the search node in longevity cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • 7.0.0
    • Cheshire-Cat
    • fts

    Description

      Build: 7.0.0 build 5219
      Test: -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale: 3

      Test Cycle 3 completed.

      Cluster config:

      ########## Cluster config ##################
      ######  kv : 11 ===== > [172.23.104.157:8091 172.23.104.69:8091 172.23.106.100:8091 172.23.108.103:8091 172.23.121.3:8091 172.23.97.119:8091 172.23.97.121:8091 172.23.97.122:8091 172.23.99.20:8091 172.23.99.21:8091 172.23.99.25:8091]  ###########
      ######  eventing : 3 ===== > [172.23.104.5:8091 172.23.123.27:8091 172.23.98.135:8091]  ###########
      ######  cbas : 3 ===== > [172.23.105.107:8091 172.23.106.188:8091 172.23.97.239:8091]  ###########
      ######  n1ql : 2 ===== > [172.23.120.245:8091 172.23.96.251:8091]  ###########
      ######  backup : 1 ===== > [172.23.123.28:8091]  ###########
      ######  index : 6 ===== > [172.23.104.137:8091 172.23.104.70:8091 172.23.121.117:8091 172.23.96.252:8091 172.23.96.253:8091 172.23.99.11:8091]  ###########
      ######  fts : 2 ===== > [172.23.104.155:8091 172.23.96.148:8091]  ###########
      

      On the node 172.23.96.148 we see around 20 % RAM usage. While on the other node 172.23.104.155 we see RAM usage at around 80% even after deleting all the fts indexes.

      Longevity run going on from 4 days. As seen in the attached screenshot the RAM usage never came down for the node: 172.23.104.155. For each Test Cycle we delete all the fts indexes at the end of the cycle. So we expect RAM usage going down at the end of the test cycle and going up at the beginning of the next test cycle as we start creating indexes. This behaviour can be seen in the node: 172.23.96.148 but not in 172.23.104.155.

      Memory profile of both search nodes are attached.

      Attachments

        1. 155-run-memory.pprof
          838 kB
          Sreekanth Sivasankaran
        2. fts_cpu_profile_148.pprof
          5 kB
          Girish Benakappa
        3. fts_cpu_profile_155.pprof
          4 kB
          Girish Benakappa
        4. fts_go_profile_148.pprof
          40 kB
          Girish Benakappa
        5. fts_go_profile_155.pprof
          111 kB
          Girish Benakappa
        6. fts_mem_profile_148.pprof.tar.gz
          6.98 MB
          Girish Benakappa
        7. fts_mem_profile_155.pprof.tar.gz
          6.98 MB
          Girish Benakappa
        8. run-memory_148.pprof
          820 kB
          Girish Benakappa
        9. Screen Shot 2021-05-26 at 10.01.09 PM.png
          92 kB
          Girish Benakappa
        10. Screen Shot 2021-05-26 at 2.55.57 PM.png
          57 kB
          Girish Benakappa
        11. Screen Shot 2021-05-26 at 2.56.25 PM.png
          73 kB
          Girish Benakappa
        12. Screenshot 2021-05-27 at 10.11.42 AM.png
          801 kB
          Sreekanth Sivasankaran
        13. Screenshot 2021-05-27 at 10.25.01 AM.png
          183 kB
          Sreekanth Sivasankaran
        14. Screen Shot 2021-05-27 at 2.27.50 PM.png
          72 kB
          Abhi Dangeti
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            abhinav Abhi Dangeti
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty