Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-1726

Client Failure continues after swap rebalance on memcached bucket on Vulcan

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None

      Description

      net client : https://github.com/couchbase/couchbase-net-client/commit/6a364d4bb3234cc5091fe239dbd26d35d7e5abff
      server : vulcan rc3 5.5.0-2938
      bucket type : memcached
      test : swap rebalance 1 node.

      Expected : no errors
      Actual : after rebalance, client failure continues till the end of the test.

        Attachments

        1. image-2018-07-06-19-30-19-855.png
          image-2018-07-06-19-30-19-855.png
          303 kB
        2. log.zip
          15.31 MB
        3. log2.zip
          15.66 MB

          Issue Links

          For Gerrit Dashboard: NCBC-1726
          # Subject Branch Project Status CR V

            Activity

            Hide
            jaekwon.park Jae Park [X] (Inactive) added a comment - - edited

            attached log2.zip which was ran on different client host. memory usage was from 2.5G~3.4G out of total memory size 8G

            Show
            jaekwon.park Jae Park [X] (Inactive) added a comment - - edited attached log2.zip which was ran on different client host. memory usage was from 2.5G~3.4G out of total memory size 8G
            Hide
            jaekwon.park Jae Park [X] (Inactive) added a comment -

            Turned out this is intermittent issue.
            I narrowed down by checking all commits by running about 10 times per each commit and found from this commit, it fails intermittently https://github.com/couchbase/couchbase-net-client/commit/44f745f83dc52f2f96af2cedf3ae42210d207a2e

            Show
            jaekwon.park Jae Park [X] (Inactive) added a comment - Turned out this is intermittent issue. I narrowed down by checking all commits by running about 10 times per each commit and found from this commit, it fails intermittently https://github.com/couchbase/couchbase-net-client/commit/44f745f83dc52f2f96af2cedf3ae42210d207a2e
            Hide
            jmorris Jeff Morris added a comment - - edited

            The issue can be traced back to this commit: NCBC-1575

            It's something to do with PLAIN SASL authentication...this may be a seperate issue, but worth investigating as related this exception:

            2018-07-03 19:37:50,308 [15] DEBUG Couchbase.IO.Services.IOServiceBase - Error when trying to execute HELO operation - KV Error: {Name="SUCCESS", Description="Success", Attributes="success"} - System.IndexOutOfRangeException: Index was outside the bounds of the array.
            at Couchbase.IO.Converters.DefaultConverter.CopyAndReverse(Byte[] src, Int32 offset, Int32 length)
            at Couchbase.IO.Converters.DefaultConverter.ToInt16(Byte[] buffer, Int32 offset, Boolean useNbo)
            at Couchbase.IO.Converters.DefaultConverter.ToInt16(Byte[] buffer, Int32 offset)
            at Couchbase.IO.Operations.Hello.GetValue()

             Thanks Jae Park [X] for isolating.

            Show
            jmorris Jeff Morris added a comment - - edited The issue can be traced back to this commit:  NCBC-1575 It's something to do with PLAIN SASL authentication...this may be a seperate issue, but worth investigating as related this exception: 2018-07-03 19:37:50,308 [15] DEBUG Couchbase.IO.Services.IOServiceBase - Error when trying to execute HELO operation - KV Error: {Name="SUCCESS", Description="Success", Attributes="success"} - System.IndexOutOfRangeException: Index was outside the bounds of the array. at Couchbase.IO.Converters.DefaultConverter.CopyAndReverse(Byte[] src, Int32 offset, Int32 length) at Couchbase.IO.Converters.DefaultConverter.ToInt16(Byte[] buffer, Int32 offset, Boolean useNbo) at Couchbase.IO.Converters.DefaultConverter.ToInt16(Byte[] buffer, Int32 offset) at Couchbase.IO.Operations.Hello.GetValue()  Thanks Jae Park [X] for isolating.
            Hide
            jmorris Jeff Morris added a comment -

            It looks like the issue exists for both SASL PLAIN and SRAM-SHA, however since SASL PLAIN is faster, it is more noticeable; you can run out of ephemeral ports temporarily. 

            Show
            jmorris Jeff Morris added a comment - It looks like the issue exists for both SASL PLAIN and SRAM-SHA, however since SASL PLAIN is faster, it is more noticeable; you can run out of ephemeral ports temporarily. 

              People

              • Assignee:
                jmorris Jeff Morris
                Reporter:
                jaekwon.park Jae Park [X] (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.