Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-33773

Crash due to race between DCP Consumer message processing and stream shutdown

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      5.5.4-4334 - centos longevity - following memcached exit observed:

      Service 'memcached' exited with status 139. Restarting. Messages:
      2019-04-11T11:19:07.686740-07:00 WARNING (DISTRICT) Slow scheduling for NON_IO task 'Adjusting hash table sizes.' on thread nonIO_worker_0. Schedule overhead: 11 s
      2019-04-11T11:19:07.686887-07:00 WARNING (HISTORY) Slow scheduling for NON_IO task 'Adjusting hash table sizes.' on thread nonIO_worker_0. Schedule overhead: 11 s
      2019-04-11T11:19:07.687010-07:00 WARNING (ITEM) Slow scheduling for NON_IO task 'Adjusting hash table sizes.' on thread nonIO_worker_0. Schedule overhead: 11 s
      2019-04-11T11:19:07.711580-07:00 WARNING (HISTORY) Slow runtime for 'Checkpoint Remover on vb 209' on thread nonIO_worker_2: 16 ms
      2019-04-11T11:19:07.722746-07:00 WARNING 136: (HISTORY) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.96.148->ns_1@172.23.96.56:HISTORY - vb:267 Got error 'no memory' while trying to process mutation with seqno:3116
      2019-04-11T11:19:07.722998-07:00 WARNING (HISTORY) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.97.239->ns_1@172.23.96.56:HISTORY - (vb 623) End stream received but no such stream for this vBucket

      Logs: http://supportal.couchbase.com/snapshot/8ff7d91dd5a771c1904307a1f2f35743::0
      Cluster: 172.23.108.103
      console: http://172.23.109.231/job/centos-systest-launcher/1767/console

      Attachments

        1. collectinfo-2019-04-12t031901-ns_1@172.23.104.164.zip
          28.21 MB
        2. collectinfo-2019-04-12t031901-ns_1@172.23.104.164.zip
          28.21 MB
        3. collectinfo-2019-04-12t031901-ns_1@172.23.104.61.zip
          22.02 MB
        4. collectinfo-2019-04-12t031901-ns_1@172.23.104.61.zip
          22.02 MB
        5. collectinfo-2019-04-12t031901-ns_1@172.23.104.67.zip
          21.39 MB
        6. collectinfo-2019-04-12t031901-ns_1@172.23.104.67.zip
          21.39 MB
        7. collectinfo-2019-04-12t031901-ns_1@172.23.104.69.zip
          18.10 MB
        8. collectinfo-2019-04-12t031901-ns_1@172.23.104.69.zip
          18.10 MB
        9. collectinfo-2019-04-12t031901-ns_1@172.23.104.87.zip
          22.70 MB
        10. collectinfo-2019-04-12t031901-ns_1@172.23.104.87.zip
          22.70 MB
        11. collectinfo-2019-04-12t031901-ns_1@172.23.104.88.zip
          23.33 MB
        12. collectinfo-2019-04-12t031901-ns_1@172.23.104.88.zip
          23.33 MB
        13. collectinfo-2019-04-12t031901-ns_1@172.23.106.188.zip
          32.92 MB
        14. collectinfo-2019-04-12t031901-ns_1@172.23.108.103.zip
          75.51 MB
        15. collectinfo-2019-04-12t031901-ns_1@172.23.108.104.zip
          57.25 MB
        16. collectinfo-2019-04-12t031901-ns_1@172.23.96.145.zip
          25.03 MB
        17. collectinfo-2019-04-12t031901-ns_1@172.23.96.56.zip
          51.07 MB
        18. collectinfo-2019-04-12t031901-ns_1@172.23.97.242.zip
          30.06 MB
        19. collectinfo-2019-04-12t031901-ns_1@172.23.99.11.zip
          31.22 MB
        20. collectinfo-2019-04-12t031901-ns_1@172.23.99.25.zip
          23.71 MB
        21. MB-33773_bug.cc
          2 kB
        22. Screen Shot 2019-04-11 at 8.17.27 PM.png
          Screen Shot 2019-04-11 at 8.17.27 PM.png
          263 kB

        Issue Links

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty