Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-1927

SDK does not work with publically addressable Kubernetes cluster

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 2.7.6
    • Fix Version/s: 2.7.9
    • Component/s: library
    • Labels:
      None

      Description

      Similar to GOCBC-431 and JVMCBC-654.

      The .NET SDK does not work with publically addressable Kubernetes clusters.

      The first problem is that it tries to resolve the hostname in the config, even if that's not possible (due to not having access to the appropriate DNS):

      Couchbase.Configuration.Server.Serialization.BootstrapException
        HResult=0x80131500
        Message=Could not bootstrap - check inner exceptions for details.
        Source=Couchbase.NetClient
        StackTrace:
         at Couchbase.Core.ClusterController.CreateBucketImpl(String bucketName, String password, IAuthenticator authenticator)
         at Couchbase.Core.ClusterController.CreateBucket(String bucketName, String password, IAuthenticator authenticator)
         at Couchbase.Cluster.OpenBucket(String bucketName, String password)
         at Couchbase.Cluster.OpenBucket(String bucketname)
         at AltAddress.Program.Main(String[] args) in C:\Users\Administrator\source\repos\AltAddress\AltAddress\Program.cs:line 27
       
      Inner Exception 1:
      TargetInvocationException: Exception has been thrown by the target of an invocation.
       
      Inner Exception 2:
      UnsupportedAddressFamilyException: http://cb-example-0000.cb-example.default.svc
      

      If I add duff entries into the hosts file then it gets past this error and seems to bootstrap correctly, however operations will randomly fail.

      Looking at why this is in the logs, it seems that the node locator is not being initialized correctly:

      19-04-16 15:26:09,192 [1] TRACE Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#109 on thread 1
      2019-04-16 15:26:09,192 [1] TRACE Couchbase.Core.Buckets.VBucketKeyMapper - Using index 77 for key Test - rev109
      2019-04-16 15:26:09,192 [1] DEBUG Couchbase.Core.VBucket - System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
         at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
         at Couchbase.Core.VBucket.LocatePrimary()
      2019-04-16 15:26:09,193 [1] DEBUG Couchbase.Core.VBucket - System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
         at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
         at Couchbase.Core.VBucket.LocatePrimary()
      

      The reason that it works sometimes is because if the vbucket locator can't find the node it needs in the vbucket map, it picks a random one - https://github.com/couchbase/couchbase-net-client/blob/2.7.6/Src/Couchbase/Core/VBucket.cs#L73.
      So if it happens to pick the 'right' node then the operation will succeed, otherwise it will fail.

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Show
            jmorris Jeff Morris added a comment - This has come up before:  https://forums.couchbase.com/t/net-core-gke-autonomous-operator-unsupportedaddressfamily
            Hide
            matt.carabine Matt Carabine added a comment - - edited

            Hey Jeff Morris - any idea when you'll get a chance to get round to this? If it'll be more than a few days I'll take the cluster down and then I can spin it back up again whenever you're ready!
            It's on AWS so just trying to be conscious of saving money.

            Show
            matt.carabine Matt Carabine added a comment - - edited Hey Jeff Morris - any idea when you'll get a chance to get round to this? If it'll be more than a few days I'll take the cluster down and then I can spin it back up again whenever you're ready! It's on AWS so just trying to be conscious of saving money.
            Hide
            jmorris Jeff Morris added a comment -

            Matt Carabine -

            My bad, Easter week and time off! I am backed up at the moment, so probably just better to power off and then Michael Goldsmith or myself can ping you (and the ticket to "in-progress" when we are ready to start working on it.  Note the fox-release is more likely to be 2.7.8 and not 2.7.7 as indicated (will be updated soon).

            We really appreciate you setting up the enviorment on AWS!

            Thanks,

            Jeff

            Show
            jmorris Jeff Morris added a comment - Matt Carabine - My bad, Easter week and time off! I am backed up at the moment, so probably just better to power off and then Michael Goldsmith or myself can ping you (and the ticket to "in-progress" when we are ready to start working on it.  Note the fox-release is more likely to be 2.7.8 and not 2.7.7 as indicated (will be updated soon). We really appreciate you setting up the enviorment on AWS! Thanks, Jeff
            Hide
            ingenthr Matt Ingenthron added a comment -

            Michael Goldsmith: Jeff told me about the SDKQE testing planned on this one that you're working toward. Quick question: do you have confidence that the change works in the env. MattC had or a similar env.?

            Show
            ingenthr Matt Ingenthron added a comment - Michael Goldsmith : Jeff told me about the SDKQE testing planned on this one that you're working toward. Quick question: do you have confidence that the change works in the env. MattC had or a similar env.?
            Hide
            matt.carabine Matt Carabine added a comment - - edited

            My environment is using public connectivity (assuming this is how the fix is being verified).

            Only place that I don't think has been tested is the NodePorts case where the alternate ports are also different too, but think this fix would cover both scenarios?

            Show
            matt.carabine Matt Carabine added a comment - - edited My environment is using public connectivity (assuming this is how the fix is being verified). Only place that I don't think has been tested is the NodePorts case where the alternate ports are also different too, but think this fix would cover both scenarios?
            Hide
            mike.goldsmith Michael Goldsmith added a comment -

            Matt Ingenthron - I used Matt Carabine's AWS test environment to verify while investigating the issue and creating the fix.

            Show
            mike.goldsmith Michael Goldsmith added a comment - Matt Ingenthron - I used Matt Carabine 's AWS test environment to verify while investigating the issue and creating the fix.
            Hide
            jmorris Jeff Morris added a comment - - edited

            Re-opening this because it triggers a regression defined in NCBC-2002; we'll need to revert this issue and come up with an alternative solution or patch it in the NCBC-2002 fix.

            Matt Carabine - can you attach a cluster map from from a Kubernetes environment with non-resolvable hostnames?

            Nevermind using attached ncbc_1927.txt.

            Show
            jmorris Jeff Morris added a comment - - edited Re-opening this because it triggers a regression defined in NCBC-2002 ; we'll need to revert this issue and come up with an alternative solution or patch it in the NCBC-2002 fix. Matt Carabine - can you attach a cluster map from from a Kubernetes environment with non-resolvable hostnames? Nevermind using attached ncbc_1927.txt.
            Hide
            jmorris Jeff Morris added a comment -

            Reverted and fixed in NCBC-2002

            Show
            jmorris Jeff Morris added a comment - Reverted and fixed in NCBC-2002

              People

              • Assignee:
                mike.goldsmith Michael Goldsmith
                Reporter:
                matt.carabine Matt Carabine
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.