Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39881

ASAN Identifying issue with Tracing functionality

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • None
    • Cheshire-Cat
    • couchbase-bucket
    • None
    • Untriaged
    • 1
    • Unknown

    Description

      During the investigation of.a potential memory corruption. Issue we ran Couchbase-Server with Address Sanitiser enabled in memcached as a toy-build ( 7.0.0-11225-enterprise)
      http://server.jenkins.couchbase.com/view/Toys/job/toy-unix-simple/1225/artifact/couchbase-server-enterprise-7.0.0-11225-centos7.x86_64.rpm

      This resulted in producing the following:

      172.23.120.201 –2020-06-05T19:00:29 (106-glibc) 3f0aac4e-cb72-4507-87359a95-088b50a0.dmp
      172.23.121.10 –2020-06-05T19:51:18 (260-glibc) 43ec1478-c321-40de-1eecd7b2-7a26b30c.dmp
      172.23.121.10 –2020-06-05T20:17:56 (260-glibc) b2c3e22c-3690-4e22-b7b9ad8e-48f89cff.dmp
      172.23.98.195 - 2020-06-05T20:20:44 (78-glibc) d36d7b5a-a14f-48bf-4d55bd91-20a67af0.dmp

      Both crashes on 172.23.121.10 are due to a ASAN issue which appears to originate from tracing code kv_engine/tracing/tracer.cc:35

      SpanId Tracer::begin(Code tracecode,
                           std::chrono::steady_clock::time_point startTime) {
          vecSpans.emplace_back(tracecode, startTime); <<<<<<< HERE
          return vecSpans.size() - 1;
      }
      

      Looking at the crashes on 172.23.98.195 and 172.23.120.201 we are hitting the same ASAN issue, which originates from tracing code kv_engine/tracing/tracer.cc:35

      Note: The machines are running different versions of glib
      172.23.104.186 - glibc-2.17-196.el7_4.2.x86_64
      172.23.120.201 - glibc-2.17-106.el7_2.1.x86_64
      172.23.120.206 - glibc-2.17-106.el7_2.1.x86_64
      172.23.121.10 - glibc-2.17-260.el7.x86_64
      172.23.98.195 - glibc-2.17-78.el7.x86_64
      172.23.98.196 - glibc-2.17-78.el7.x86_64

      Logs

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/bufferevent_OoO/collectinfo-2020-06-06T033130-ns_1%40172.23.104.186.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/bufferevent_OoO/collectinfo-2020-06-06T033130-ns_1%40172.23.120.201.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/bufferevent_OoO/collectinfo-2020-06-06T033130-ns_1%40172.23.120.206.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/bufferevent_OoO/collectinfo-2020-06-06T033130-ns_1%40172.23.121.10.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/bufferevent_OoO/collectinfo-2020-06-06T033130-ns_1%40172.23.98.195.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/bufferevent_OoO/collectinfo-2020-06-06T033130-ns_1%40172.23.98.196.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              owend Daniel Owen
              owend Daniel Owen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty