Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43070

[System Test] : query service (cbq-engine) not responding

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 7.0.0
    • Cheshire-Cat
    • query

    Description

      Build : 7.0.0-3875
      Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll.yml -scope tests/integration/cheshirecat/scope_cheshirecat.yml
      Scale : 3
      Iteration : 1 (day 1)

      In the longevity test with collection support, there are steps to run N1QL queries continuously against collections. Concurrently, queries are also run on buckets. And there are also threads that run in parallel that will create indexes on collections of other buckets.

      Both the query nodes get hung after a while and stop responding. Tried accessing QWB and cbq-engine - it does not respond. Same result issuing a query via REST.

      Attached are the CPU/Heap/Memory profiles and active/completed requests and goroutine dumps from both the query nodes - 172.23.96.251 & 172.23.96.252. Also have specified the location of the cbcollect logs.

      Cluster Config :
      Eventing : 3 ===== > [172.23.104.137:8091 172.23.121.3:8091 172.23.123.27:8091]
      FTS : 2 ===== > [172.23.104.157:8091 172.23.120.245:8091]
      KV : 11 ===== > [172.23.104.5:8091 172.23.108.103:8091 172.23.96.148:8091 172.23.97.119:8091 172.23.97.121:8091 172.23.97.122:8091 172.23.97.239:8091 172.23.98.135:8091 172.23.99.20:8091 172.23.99.21:8091 172.23.99.25:8091]
      Index : 6 ===== > [172.23.104.67:8091 172.23.104.69:8091 172.23.104.70:8091 172.23.123.28:8091 172.23.96.253:8091 172.23.96.95:8091]
      CBAS : 3 ===== > [172.23.106.188:8091 172.23.97.242:8091 172.23.99.11:8091]
      N1QL : 2 ===== > [172.23.96.251:8091 172.23.96.252:8091]

      Attachments

        1. heap_252.prof
          193 kB
        2. query_active_requests_251.json
          23 kB
        3. heap_251.prof
          182 kB
        4. query_active_requests_252.json
          18 kB
        5. query_cpu_profile_251.pprof
          43 kB
        6. query_completed_requests_251.json
          182 kB
        7. query_cpu_profile_252.pprof
          42 kB
        8. query_completed_requests_252.json
          208 kB
        9. query_252.mprof
          24.14 MB
        10. goroutine_dump_252.txt
          28.83 MB
        11. goroutine_dump_251.txt
          26.87 MB
        12. query_251.mprof
          23.40 MB
        13. query_dumps0109.zip
          10.30 MB

        Issue Links

          depends on

          Bug - A problem which impairs or prevents the functions of the product. MB-43486 Leak due to verbose logging MultiScanInternal()

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-43070
          # Subject Branch Project Status CR V
          141696,3 MB-43070 stall clearing stats storage master query-ee Status: MERGED +2 +1
          141891,2 MB-43070 only have one CBO stats cleaner master query Status: ABANDONED 0 0
          142929,2 MB-43070 only have one session cleaning the dict cache on scope drop master query Status: MERGED +2 +1
          205023,2 MB-60614 Avoid asynchronous removal of CBO stats when possible trinity query Status: MERGED +2 +1
          205024,2 MB-60614 Avoid asynchronous removal of CBO stats when possible trinity query-ee Status: MERGED +2 +1

          Activity

            People

              marco.greco Marco Greco (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty