Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-1726

Client Failure continues after swap rebalance on memcached bucket on Vulcan

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.6.0
    • None
    • None
    • 1

    Description

      net client : https://github.com/couchbase/couchbase-net-client/commit/6a364d4bb3234cc5091fe239dbd26d35d7e5abff
      server : vulcan rc3 5.5.0-2938
      bucket type : memcached
      test : swap rebalance 1 node.

      Expected : no errors
      Actual : after rebalance, client failure continues till the end of the test.

      Attachments

        1. image-2018-07-06-19-30-19-855.png
          image-2018-07-06-19-30-19-855.png
          303 kB
        2. log.zip
          15.31 MB
        3. log2.zip
          15.66 MB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            jaekwon.park Jae Park [X] (Inactive) added a comment - - edited

            attached log2.zip which was ran on different client host. memory usage was from 2.5G~3.4G out of total memory size 8G

            jaekwon.park Jae Park [X] (Inactive) added a comment - - edited attached log2.zip which was ran on different client host. memory usage was from 2.5G~3.4G out of total memory size 8G

            Turned out this is intermittent issue.
            I narrowed down by checking all commits by running about 10 times per each commit and found from this commit, it fails intermittently https://github.com/couchbase/couchbase-net-client/commit/44f745f83dc52f2f96af2cedf3ae42210d207a2e

            jaekwon.park Jae Park [X] (Inactive) added a comment - Turned out this is intermittent issue. I narrowed down by checking all commits by running about 10 times per each commit and found from this commit, it fails intermittently https://github.com/couchbase/couchbase-net-client/commit/44f745f83dc52f2f96af2cedf3ae42210d207a2e
            jmorris Jeff Morris added a comment - - edited

            The issue can be traced back to this commit: NCBC-1575

            It's something to do with PLAIN SASL authentication...this may be a seperate issue, but worth investigating as related this exception:

            2018-07-03 19:37:50,308 [15] DEBUG Couchbase.IO.Services.IOServiceBase - Error when trying to execute HELO operation - KV Error: {Name="SUCCESS", Description="Success", Attributes="success"} - System.IndexOutOfRangeException: Index was outside the bounds of the array.
            at Couchbase.IO.Converters.DefaultConverter.CopyAndReverse(Byte[] src, Int32 offset, Int32 length)
            at Couchbase.IO.Converters.DefaultConverter.ToInt16(Byte[] buffer, Int32 offset, Boolean useNbo)
            at Couchbase.IO.Converters.DefaultConverter.ToInt16(Byte[] buffer, Int32 offset)
            at Couchbase.IO.Operations.Hello.GetValue()

             Thanks Jae Park [X] for isolating.

            jmorris Jeff Morris added a comment - - edited The issue can be traced back to this commit:  NCBC-1575 It's something to do with PLAIN SASL authentication...this may be a seperate issue, but worth investigating as related this exception: 2018-07-03 19:37:50,308 [15] DEBUG Couchbase.IO.Services.IOServiceBase - Error when trying to execute HELO operation - KV Error: {Name="SUCCESS", Description="Success", Attributes="success"} - System.IndexOutOfRangeException: Index was outside the bounds of the array. at Couchbase.IO.Converters.DefaultConverter.CopyAndReverse(Byte[] src, Int32 offset, Int32 length) at Couchbase.IO.Converters.DefaultConverter.ToInt16(Byte[] buffer, Int32 offset, Boolean useNbo) at Couchbase.IO.Converters.DefaultConverter.ToInt16(Byte[] buffer, Int32 offset) at Couchbase.IO.Operations.Hello.GetValue()  Thanks Jae Park [X] for isolating.
            jmorris Jeff Morris added a comment -

            It looks like the issue exists for both SASL PLAIN and SRAM-SHA, however since SASL PLAIN is faster, it is more noticeable; you can run out of ephemeral ports temporarily. 

            jmorris Jeff Morris added a comment - It looks like the issue exists for both SASL PLAIN and SRAM-SHA, however since SASL PLAIN is faster, it is more noticeable; you can run out of ephemeral ports temporarily. 

            People

              jmorris Jeff Morris
              jaekwon.park Jae Park [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty