Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47457

FTS is broken with Alternate Addresses set

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes

    Description

      Summary

      FTS incorrectly uses the alternate addresses set for a node for DCP streams.

      This means in the worst (and common) case where the alternate address is not reachable within the cluster (e.g. it's in Kubernetes and it's a Load Balancer for an individual pod, a fairly common deployment scenario), FTS is completely broken, while indexes can be created they are never built and cannot be searched.

      In the best case where the cluster is not deployed in Kubernetes but is using Alternate Addresses for things like XDCR, the DCP traffic is being incorrectly routed out over the public internet rather than the private network as it should be.

      This is a regression, FTS works fine in this environment on 6.6.x.

      Steps to Reproduce

      1. Spin up the latest RC (I used Docker here):

        docker run -d --name cc-rc -p 8091-8097:8091-8097 registry.gitlab.com/cb-vanilla/server:7.0.0-5302
        

      2. Setup the cluster with FTS enabled. Ensure that you set the hostname of the node to its IP and NOT 127.0.0.1 at this step. I also enabled node-to-node encryption, not sure if this is required for reproduction.
      3. Set an alternate address for your node, you can set this to any URL that will resolve but will not be accessible, I just used a Couchbase Cloud public hostname:

        /opt/couchbase/bin/couchbase-cli setting-alternate-address --cluster localhost --username Administrator --password password --set --hostname cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com --node 172.17.0.2
        

      4. Restart the cbft process
      5. Create an Index in the UI
      6. Try to search the Index

      Expected Result

      The index search completes successfully

      Actual Result

      Investigation

      Logs show that FTS is trying to use the external address:

      2021-07-16T11:10:57.457+00:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:test UserAgent:matt_7adf0964fce22708_4c1c5584 UseTLS:false NetworkType: Auth:0x1d368d0 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}
       
      2021-07-16T11:11:02.517+00:00 [WARN] (GOCBCORE) Pipeline Client 0xc00069ee40 failed to bootstrap: dial tcp 18.210.140.11:11210: i/o timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
      

      Note that 18.210.140.11 is what cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com resolves to:

      nslookup cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
      Server:         8.8.8.8
      Address:        8.8.8.8#53
       
      Non-authoritative answer:
      Name:   cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
      Address: 18.210.140.11
      

      Also strangely it's trying to use 11210, but node-to-node encryption is enabled, which I would have expected meant that it would try to use encrypted ports.

      Logs

      https://cb-engineering.s3.amazonaws.com/MB-47457/collectinfo-2021-07-19T133501-ns_1%40172.17.0.2.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            matt.carabine Matt Carabine created issue -
            matt.carabine Matt Carabine made changes -
            Field Original Value New Value
            Link This issue blocks CBSE-10439 [ CBSE-10439 ]
            matt.carabine Matt Carabine made changes -
            Description h2. Summary

            FTS incorrectly uses the alternate addresses set for a node for DCP streams.

            This means in the worst case where the alternate address is not reachable within the cluster (e.g. it's in Kubernetes and it's a Load Balancer for an individual pod, a fairly common deployment scenario), FTS is completely broken, while indexes can be created they are never built and cannot be searched.

            In the best case, the DCP traffic is being incorrectly routed out over the public internet rather than the private network as it should be.

            h2. Steps to Reproduce

            # Spin up the latest RC (I used Docker here):
            {noformat}
            docker run -d --name cc-rc -p 8091-8097:8091-8097 registry.gitlab.com/cb-vanilla/server:7.0.0-5302
            {noformat}
            # Setup the cluster with FTS enabled. *Ensure that you set the hostname of the node to its IP and NOT 127.0.0.1 at this step*.
            # Set an alternate address for your node, you can set this to any URL that will resolve but will not be accessible, I just used a Couchbase Cloud public hostname: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-alternate-address --cluster localhost --username Administrator --password password --set --hostname cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com --node 172.17.0.2
            {noformat}
            # Restart the cbft process
            # Create an Index in the UI
            # Try to search the Index

            h2. Expected Result

            The index search completes successfully

            h2. Actual Result

             !image-2021-07-16-12-22-58-756.png|thumbnail!

            h2. Investigation

            Logs show that FTS is trying to use the external address: {noformat}
            2021-07-16T11:10:57.457+00:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:test UserAgent:matt_7adf0964fce22708_4c1c5584 UseTLS:false NetworkType: Auth:0x1d368d0 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}

            2021-07-16T11:11:02.517+00:00 [WARN] (GOCBCORE) Pipeline Client 0xc00069ee40 failed to bootstrap: dial tcp 18.210.140.11:11210: i/o timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
            {noformat}

            Note that 18.210.140.11 is what cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com resolves to:
            {noformat}
            nslookup cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Server: 8.8.8.8
            Address: 8.8.8.8#53

            Non-authoritative answer:
            Name: cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Address: 18.210.140.11
            {noformat}

            Also strangely it's trying to use 11210, but node-to-node encryption is enabled, which I would have expected meant that it would try to use encrypted ports.

            h2. Logs

            Haven't collected logs as this issue is trivially reproducible with a 100% success rate, if really required I can collect and attach.
            h2. Summary

            FTS incorrectly uses the alternate addresses set for a node for DCP streams.

            This means in the worst case where the alternate address is not reachable within the cluster (e.g. it's in Kubernetes and it's a Load Balancer for an individual pod, a fairly common deployment scenario), FTS is completely broken, while indexes can be created they are never built and cannot be searched.

            In the best case, the DCP traffic is being incorrectly routed out over the public internet rather than the private network as it should be.

            h2. Steps to Reproduce

            # Spin up the latest RC (I used Docker here):
            {noformat}
            docker run -d --name cc-rc -p 8091-8097:8091-8097 registry.gitlab.com/cb-vanilla/server:7.0.0-5302
            {noformat}
            # Setup the cluster with FTS enabled. *Ensure that you set the hostname of the node to its IP and NOT 127.0.0.1 at this step*. I also enabled node-to-node encryption, not sure if this is _required_ for reproduction.
            # Set an alternate address for your node, you can set this to any URL that will resolve but will not be accessible, I just used a Couchbase Cloud public hostname: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-alternate-address --cluster localhost --username Administrator --password password --set --hostname cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com --node 172.17.0.2
            {noformat}
            # Restart the cbft process
            # Create an Index in the UI
            # Try to search the Index

            h2. Expected Result

            The index search completes successfully

            h2. Actual Result

             !image-2021-07-16-12-22-58-756.png|thumbnail!

            h2. Investigation

            Logs show that FTS is trying to use the external address: {noformat}
            2021-07-16T11:10:57.457+00:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:test UserAgent:matt_7adf0964fce22708_4c1c5584 UseTLS:false NetworkType: Auth:0x1d368d0 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}

            2021-07-16T11:11:02.517+00:00 [WARN] (GOCBCORE) Pipeline Client 0xc00069ee40 failed to bootstrap: dial tcp 18.210.140.11:11210: i/o timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
            {noformat}

            Note that 18.210.140.11 is what cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com resolves to:
            {noformat}
            nslookup cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Server: 8.8.8.8
            Address: 8.8.8.8#53

            Non-authoritative answer:
            Name: cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Address: 18.210.140.11
            {noformat}

            Also strangely it's trying to use 11210, but node-to-node encryption is enabled, which I would have expected meant that it would try to use encrypted ports.

            h2. Logs

            Haven't collected logs as this issue is trivially reproducible with a 100% success rate, if really required I can collect and attach.
            matt.carabine Matt Carabine made changes -
            Is this a Regression? Unknown [ 10452 ] Yes [ 10450 ]
            matt.carabine Matt Carabine made changes -
            Description h2. Summary

            FTS incorrectly uses the alternate addresses set for a node for DCP streams.

            This means in the worst case where the alternate address is not reachable within the cluster (e.g. it's in Kubernetes and it's a Load Balancer for an individual pod, a fairly common deployment scenario), FTS is completely broken, while indexes can be created they are never built and cannot be searched.

            In the best case, the DCP traffic is being incorrectly routed out over the public internet rather than the private network as it should be.

            h2. Steps to Reproduce

            # Spin up the latest RC (I used Docker here):
            {noformat}
            docker run -d --name cc-rc -p 8091-8097:8091-8097 registry.gitlab.com/cb-vanilla/server:7.0.0-5302
            {noformat}
            # Setup the cluster with FTS enabled. *Ensure that you set the hostname of the node to its IP and NOT 127.0.0.1 at this step*. I also enabled node-to-node encryption, not sure if this is _required_ for reproduction.
            # Set an alternate address for your node, you can set this to any URL that will resolve but will not be accessible, I just used a Couchbase Cloud public hostname: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-alternate-address --cluster localhost --username Administrator --password password --set --hostname cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com --node 172.17.0.2
            {noformat}
            # Restart the cbft process
            # Create an Index in the UI
            # Try to search the Index

            h2. Expected Result

            The index search completes successfully

            h2. Actual Result

             !image-2021-07-16-12-22-58-756.png|thumbnail!

            h2. Investigation

            Logs show that FTS is trying to use the external address: {noformat}
            2021-07-16T11:10:57.457+00:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:test UserAgent:matt_7adf0964fce22708_4c1c5584 UseTLS:false NetworkType: Auth:0x1d368d0 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}

            2021-07-16T11:11:02.517+00:00 [WARN] (GOCBCORE) Pipeline Client 0xc00069ee40 failed to bootstrap: dial tcp 18.210.140.11:11210: i/o timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
            {noformat}

            Note that 18.210.140.11 is what cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com resolves to:
            {noformat}
            nslookup cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Server: 8.8.8.8
            Address: 8.8.8.8#53

            Non-authoritative answer:
            Name: cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Address: 18.210.140.11
            {noformat}

            Also strangely it's trying to use 11210, but node-to-node encryption is enabled, which I would have expected meant that it would try to use encrypted ports.

            h2. Logs

            Haven't collected logs as this issue is trivially reproducible with a 100% success rate, if really required I can collect and attach.
            h2. Summary

            FTS incorrectly uses the alternate addresses set for a node for DCP streams.

            This means in the worst case where the alternate address is not reachable within the cluster (e.g. it's in Kubernetes and it's a Load Balancer for an individual pod, a fairly common deployment scenario), FTS is completely broken, while indexes can be created they are never built and cannot be searched.

            In the best case, the DCP traffic is being incorrectly routed out over the public internet rather than the private network as it should be.

            This is a regression, FTS works fine in this environment on 6.6.x.

            h2. Steps to Reproduce

            # Spin up the latest RC (I used Docker here):
            {noformat}
            docker run -d --name cc-rc -p 8091-8097:8091-8097 registry.gitlab.com/cb-vanilla/server:7.0.0-5302
            {noformat}
            # Setup the cluster with FTS enabled. *Ensure that you set the hostname of the node to its IP and NOT 127.0.0.1 at this step*. I also enabled node-to-node encryption, not sure if this is _required_ for reproduction.
            # Set an alternate address for your node, you can set this to any URL that will resolve but will not be accessible, I just used a Couchbase Cloud public hostname: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-alternate-address --cluster localhost --username Administrator --password password --set --hostname cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com --node 172.17.0.2
            {noformat}
            # Restart the cbft process
            # Create an Index in the UI
            # Try to search the Index

            h2. Expected Result

            The index search completes successfully

            h2. Actual Result

             !image-2021-07-16-12-22-58-756.png|thumbnail!

            h2. Investigation

            Logs show that FTS is trying to use the external address: {noformat}
            2021-07-16T11:10:57.457+00:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:test UserAgent:matt_7adf0964fce22708_4c1c5584 UseTLS:false NetworkType: Auth:0x1d368d0 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}

            2021-07-16T11:11:02.517+00:00 [WARN] (GOCBCORE) Pipeline Client 0xc00069ee40 failed to bootstrap: dial tcp 18.210.140.11:11210: i/o timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
            {noformat}

            Note that 18.210.140.11 is what cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com resolves to:
            {noformat}
            nslookup cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Server: 8.8.8.8
            Address: 8.8.8.8#53

            Non-authoritative answer:
            Name: cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Address: 18.210.140.11
            {noformat}

            Also strangely it's trying to use 11210, but node-to-node encryption is enabled, which I would have expected meant that it would try to use encrypted ports.

            h2. Logs

            Haven't collected logs as this issue is trivially reproducible with a 100% success rate, if really required I can collect and attach.
            matt.carabine Matt Carabine made changes -
            Description h2. Summary

            FTS incorrectly uses the alternate addresses set for a node for DCP streams.

            This means in the worst case where the alternate address is not reachable within the cluster (e.g. it's in Kubernetes and it's a Load Balancer for an individual pod, a fairly common deployment scenario), FTS is completely broken, while indexes can be created they are never built and cannot be searched.

            In the best case, the DCP traffic is being incorrectly routed out over the public internet rather than the private network as it should be.

            This is a regression, FTS works fine in this environment on 6.6.x.

            h2. Steps to Reproduce

            # Spin up the latest RC (I used Docker here):
            {noformat}
            docker run -d --name cc-rc -p 8091-8097:8091-8097 registry.gitlab.com/cb-vanilla/server:7.0.0-5302
            {noformat}
            # Setup the cluster with FTS enabled. *Ensure that you set the hostname of the node to its IP and NOT 127.0.0.1 at this step*. I also enabled node-to-node encryption, not sure if this is _required_ for reproduction.
            # Set an alternate address for your node, you can set this to any URL that will resolve but will not be accessible, I just used a Couchbase Cloud public hostname: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-alternate-address --cluster localhost --username Administrator --password password --set --hostname cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com --node 172.17.0.2
            {noformat}
            # Restart the cbft process
            # Create an Index in the UI
            # Try to search the Index

            h2. Expected Result

            The index search completes successfully

            h2. Actual Result

             !image-2021-07-16-12-22-58-756.png|thumbnail!

            h2. Investigation

            Logs show that FTS is trying to use the external address: {noformat}
            2021-07-16T11:10:57.457+00:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:test UserAgent:matt_7adf0964fce22708_4c1c5584 UseTLS:false NetworkType: Auth:0x1d368d0 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}

            2021-07-16T11:11:02.517+00:00 [WARN] (GOCBCORE) Pipeline Client 0xc00069ee40 failed to bootstrap: dial tcp 18.210.140.11:11210: i/o timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
            {noformat}

            Note that 18.210.140.11 is what cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com resolves to:
            {noformat}
            nslookup cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Server: 8.8.8.8
            Address: 8.8.8.8#53

            Non-authoritative answer:
            Name: cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Address: 18.210.140.11
            {noformat}

            Also strangely it's trying to use 11210, but node-to-node encryption is enabled, which I would have expected meant that it would try to use encrypted ports.

            h2. Logs

            Haven't collected logs as this issue is trivially reproducible with a 100% success rate, if really required I can collect and attach.
            h2. Summary

            FTS incorrectly uses the alternate addresses set for a node for DCP streams.

            This means in the worst (and common) case where the alternate address is not reachable within the cluster (e.g. it's in Kubernetes and it's a Load Balancer for an individual pod, a fairly common deployment scenario), FTS is completely broken, while indexes can be created they are never built and cannot be searched.

            In the best case, the DCP traffic is being incorrectly routed out over the public internet rather than the private network as it should be.

            This is a regression, FTS works fine in this environment on 6.6.x.

            h2. Steps to Reproduce

            # Spin up the latest RC (I used Docker here):
            {noformat}
            docker run -d --name cc-rc -p 8091-8097:8091-8097 registry.gitlab.com/cb-vanilla/server:7.0.0-5302
            {noformat}
            # Setup the cluster with FTS enabled. *Ensure that you set the hostname of the node to its IP and NOT 127.0.0.1 at this step*. I also enabled node-to-node encryption, not sure if this is _required_ for reproduction.
            # Set an alternate address for your node, you can set this to any URL that will resolve but will not be accessible, I just used a Couchbase Cloud public hostname: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-alternate-address --cluster localhost --username Administrator --password password --set --hostname cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com --node 172.17.0.2
            {noformat}
            # Restart the cbft process
            # Create an Index in the UI
            # Try to search the Index

            h2. Expected Result

            The index search completes successfully

            h2. Actual Result

             !image-2021-07-16-12-22-58-756.png|thumbnail!

            h2. Investigation

            Logs show that FTS is trying to use the external address: {noformat}
            2021-07-16T11:10:57.457+00:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:test UserAgent:matt_7adf0964fce22708_4c1c5584 UseTLS:false NetworkType: Auth:0x1d368d0 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}

            2021-07-16T11:11:02.517+00:00 [WARN] (GOCBCORE) Pipeline Client 0xc00069ee40 failed to bootstrap: dial tcp 18.210.140.11:11210: i/o timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
            {noformat}

            Note that 18.210.140.11 is what cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com resolves to:
            {noformat}
            nslookup cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Server: 8.8.8.8
            Address: 8.8.8.8#53

            Non-authoritative answer:
            Name: cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Address: 18.210.140.11
            {noformat}

            Also strangely it's trying to use 11210, but node-to-node encryption is enabled, which I would have expected meant that it would try to use encrypted ports.

            h2. Logs

            Haven't collected logs as this issue is trivially reproducible with a 100% success rate, if really required I can collect and attach.
            abhinav Abhinav Dangeti made changes -
            Assignee Keshav Murthy [ keshav ] Abhinav Dangeti [ abhinav ]

            Matt Carabine I'll need to look into the alternate address resolution.

            As for the bit on node-to-node encryption, the log snippet you've shared regarding the "agent" connecting to the port 8091 is used for obtaining stats and the collections manifest. The DCP agent which is setup eventually will use TLS.

            abhinav Abhinav Dangeti added a comment - Matt Carabine  I'll need to look into the alternate address resolution. As for the bit on node-to-node encryption, the log snippet you've shared regarding the "agent" connecting to the port 8091 is used for obtaining stats and the collections manifest. The DCP agent which is setup eventually will use TLS.

            Abhinav Dangeti I was more referencing the dial tcp 18.210.140.11:11210: i/o timeout part, which is using the plain text KV port

            matt.carabine Matt Carabine added a comment - Abhinav Dangeti I was more referencing the dial tcp 18.210.140.11:11210: i/o timeout part, which is using the plain text KV port

            Ya I mean - when the agent connects to the SSL port 18091, the pipeline client is directed to the encrypted KV port.

            abhinav Abhinav Dangeti added a comment - Ya I mean - when the agent connects to the SSL port 18091, the pipeline client is directed to the encrypted KV port.
            abhinav Abhinav Dangeti made changes -
            Fix Version/s 7.0.1 [ 17104 ]

            Charles Dixon, will need some guidance from you on how FTS can channel the gocbcore agents and DCPagents to use alternate addresses where applicable.

            abhinav Abhinav Dangeti added a comment - Charles Dixon , will need some guidance from you on how FTS can channel the gocbcore agents and DCPagents to use alternate addresses where applicable.
            matt.carabine Matt Carabine made changes -
            Description h2. Summary

            FTS incorrectly uses the alternate addresses set for a node for DCP streams.

            This means in the worst (and common) case where the alternate address is not reachable within the cluster (e.g. it's in Kubernetes and it's a Load Balancer for an individual pod, a fairly common deployment scenario), FTS is completely broken, while indexes can be created they are never built and cannot be searched.

            In the best case, the DCP traffic is being incorrectly routed out over the public internet rather than the private network as it should be.

            This is a regression, FTS works fine in this environment on 6.6.x.

            h2. Steps to Reproduce

            # Spin up the latest RC (I used Docker here):
            {noformat}
            docker run -d --name cc-rc -p 8091-8097:8091-8097 registry.gitlab.com/cb-vanilla/server:7.0.0-5302
            {noformat}
            # Setup the cluster with FTS enabled. *Ensure that you set the hostname of the node to its IP and NOT 127.0.0.1 at this step*. I also enabled node-to-node encryption, not sure if this is _required_ for reproduction.
            # Set an alternate address for your node, you can set this to any URL that will resolve but will not be accessible, I just used a Couchbase Cloud public hostname: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-alternate-address --cluster localhost --username Administrator --password password --set --hostname cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com --node 172.17.0.2
            {noformat}
            # Restart the cbft process
            # Create an Index in the UI
            # Try to search the Index

            h2. Expected Result

            The index search completes successfully

            h2. Actual Result

             !image-2021-07-16-12-22-58-756.png|thumbnail!

            h2. Investigation

            Logs show that FTS is trying to use the external address: {noformat}
            2021-07-16T11:10:57.457+00:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:test UserAgent:matt_7adf0964fce22708_4c1c5584 UseTLS:false NetworkType: Auth:0x1d368d0 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}

            2021-07-16T11:11:02.517+00:00 [WARN] (GOCBCORE) Pipeline Client 0xc00069ee40 failed to bootstrap: dial tcp 18.210.140.11:11210: i/o timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
            {noformat}

            Note that 18.210.140.11 is what cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com resolves to:
            {noformat}
            nslookup cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Server: 8.8.8.8
            Address: 8.8.8.8#53

            Non-authoritative answer:
            Name: cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Address: 18.210.140.11
            {noformat}

            Also strangely it's trying to use 11210, but node-to-node encryption is enabled, which I would have expected meant that it would try to use encrypted ports.

            h2. Logs

            Haven't collected logs as this issue is trivially reproducible with a 100% success rate, if really required I can collect and attach.
            h2. Summary

            FTS incorrectly uses the alternate addresses set for a node for DCP streams.

            This means in the worst (and common) case where the alternate address is not reachable within the cluster (e.g. it's in Kubernetes and it's a Load Balancer for an individual pod, a fairly common deployment scenario), FTS is completely broken, while indexes can be created they are never built and cannot be searched.

            In the best case where the cluster is not deployed in Kubernetes but is using Alternate Addresses for things like XDCR, the DCP traffic is being incorrectly routed out over the public internet rather than the private network as it should be.

            This is a regression, FTS works fine in this environment on 6.6.x.

            h2. Steps to Reproduce

            # Spin up the latest RC (I used Docker here):
            {noformat}
            docker run -d --name cc-rc -p 8091-8097:8091-8097 registry.gitlab.com/cb-vanilla/server:7.0.0-5302
            {noformat}
            # Setup the cluster with FTS enabled. *Ensure that you set the hostname of the node to its IP and NOT 127.0.0.1 at this step*. I also enabled node-to-node encryption, not sure if this is _required_ for reproduction.
            # Set an alternate address for your node, you can set this to any URL that will resolve but will not be accessible, I just used a Couchbase Cloud public hostname: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-alternate-address --cluster localhost --username Administrator --password password --set --hostname cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com --node 172.17.0.2
            {noformat}
            # Restart the cbft process
            # Create an Index in the UI
            # Try to search the Index

            h2. Expected Result

            The index search completes successfully

            h2. Actual Result

             !image-2021-07-16-12-22-58-756.png|thumbnail!

            h2. Investigation

            Logs show that FTS is trying to use the external address: {noformat}
            2021-07-16T11:10:57.457+00:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:test UserAgent:matt_7adf0964fce22708_4c1c5584 UseTLS:false NetworkType: Auth:0x1d368d0 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}

            2021-07-16T11:11:02.517+00:00 [WARN] (GOCBCORE) Pipeline Client 0xc00069ee40 failed to bootstrap: dial tcp 18.210.140.11:11210: i/o timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
            {noformat}

            Note that 18.210.140.11 is what cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com resolves to:
            {noformat}
            nslookup cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Server: 8.8.8.8
            Address: 8.8.8.8#53

            Non-authoritative answer:
            Name: cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Address: 18.210.140.11
            {noformat}

            Also strangely it's trying to use 11210, but node-to-node encryption is enabled, which I would have expected meant that it would try to use encrypted ports.

            h2. Logs

            Haven't collected logs as this issue is trivially reproducible with a 100% success rate, if really required I can collect and attach.
            matt.carabine Matt Carabine made changes -
            Description h2. Summary

            FTS incorrectly uses the alternate addresses set for a node for DCP streams.

            This means in the worst (and common) case where the alternate address is not reachable within the cluster (e.g. it's in Kubernetes and it's a Load Balancer for an individual pod, a fairly common deployment scenario), FTS is completely broken, while indexes can be created they are never built and cannot be searched.

            In the best case where the cluster is not deployed in Kubernetes but is using Alternate Addresses for things like XDCR, the DCP traffic is being incorrectly routed out over the public internet rather than the private network as it should be.

            This is a regression, FTS works fine in this environment on 6.6.x.

            h2. Steps to Reproduce

            # Spin up the latest RC (I used Docker here):
            {noformat}
            docker run -d --name cc-rc -p 8091-8097:8091-8097 registry.gitlab.com/cb-vanilla/server:7.0.0-5302
            {noformat}
            # Setup the cluster with FTS enabled. *Ensure that you set the hostname of the node to its IP and NOT 127.0.0.1 at this step*. I also enabled node-to-node encryption, not sure if this is _required_ for reproduction.
            # Set an alternate address for your node, you can set this to any URL that will resolve but will not be accessible, I just used a Couchbase Cloud public hostname: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-alternate-address --cluster localhost --username Administrator --password password --set --hostname cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com --node 172.17.0.2
            {noformat}
            # Restart the cbft process
            # Create an Index in the UI
            # Try to search the Index

            h2. Expected Result

            The index search completes successfully

            h2. Actual Result

             !image-2021-07-16-12-22-58-756.png|thumbnail!

            h2. Investigation

            Logs show that FTS is trying to use the external address: {noformat}
            2021-07-16T11:10:57.457+00:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:test UserAgent:matt_7adf0964fce22708_4c1c5584 UseTLS:false NetworkType: Auth:0x1d368d0 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}

            2021-07-16T11:11:02.517+00:00 [WARN] (GOCBCORE) Pipeline Client 0xc00069ee40 failed to bootstrap: dial tcp 18.210.140.11:11210: i/o timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
            {noformat}

            Note that 18.210.140.11 is what cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com resolves to:
            {noformat}
            nslookup cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Server: 8.8.8.8
            Address: 8.8.8.8#53

            Non-authoritative answer:
            Name: cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Address: 18.210.140.11
            {noformat}

            Also strangely it's trying to use 11210, but node-to-node encryption is enabled, which I would have expected meant that it would try to use encrypted ports.

            h2. Logs

            Haven't collected logs as this issue is trivially reproducible with a 100% success rate, if really required I can collect and attach.
            h2. Summary

            FTS incorrectly uses the alternate addresses set for a node for DCP streams.

            This means in the worst (and common) case where the alternate address is not reachable within the cluster (e.g. it's in Kubernetes and it's a Load Balancer for an individual pod, a fairly common deployment scenario), FTS is completely broken, while indexes can be created they are never built and cannot be searched.

            In the best case where the cluster is not deployed in Kubernetes but is using Alternate Addresses for things like XDCR, the DCP traffic is being incorrectly routed out over the public internet rather than the private network as it should be.

            This is a regression, FTS works fine in this environment on 6.6.x.

            h2. Steps to Reproduce

            # Spin up the latest RC (I used Docker here):
            {noformat}
            docker run -d --name cc-rc -p 8091-8097:8091-8097 registry.gitlab.com/cb-vanilla/server:7.0.0-5302
            {noformat}
            # Setup the cluster with FTS enabled. *Ensure that you set the hostname of the node to its IP and NOT 127.0.0.1 at this step*. I also enabled node-to-node encryption, not sure if this is _required_ for reproduction.
            # Set an alternate address for your node, you can set this to any URL that will resolve but will not be accessible, I just used a Couchbase Cloud public hostname: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-alternate-address --cluster localhost --username Administrator --password password --set --hostname cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com --node 172.17.0.2
            {noformat}
            # Restart the cbft process
            # Create an Index in the UI
            # Try to search the Index

            h2. Expected Result

            The index search completes successfully

            h2. Actual Result

             !image-2021-07-16-12-22-58-756.png|thumbnail!

            h2. Investigation

            Logs show that FTS is trying to use the external address: {noformat}
            2021-07-16T11:10:57.457+00:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:test UserAgent:matt_7adf0964fce22708_4c1c5584 UseTLS:false NetworkType: Auth:0x1d368d0 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}

            2021-07-16T11:11:02.517+00:00 [WARN] (GOCBCORE) Pipeline Client 0xc00069ee40 failed to bootstrap: dial tcp 18.210.140.11:11210: i/o timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
            {noformat}

            Note that 18.210.140.11 is what cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com resolves to:
            {noformat}
            nslookup cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Server: 8.8.8.8
            Address: 8.8.8.8#53

            Non-authoritative answer:
            Name: cb-0001.76aad0f6-de8a-46d8-9794-47df1b10f91f.dataplane.nonprod-project-avengers.com
            Address: 18.210.140.11
            {noformat}

            Also strangely it's trying to use 11210, but node-to-node encryption is enabled, which I would have expected meant that it would try to use encrypted ports.

            h2. Logs

            https://cb-engineering.s3.amazonaws.com/MB-47457/collectinfo-2021-07-19T133501-ns_1%40172.17.0.2.zip

            Synced up with Brett Lawson on this. A fix to the agents config should fix the situation here -

             

            abhinav Abhinav Dangeti added a comment - Synced up with Brett Lawson  on this. A fix to the agents config should fix the situation here - http://review.couchbase.org/c/cbgt/+/157795  
            wayne Wayne Siu made changes -
            Link This issue blocks MB-46308 [ MB-46308 ]
            wayne Wayne Siu made changes -
            Link This issue blocks MB-47469 [ MB-47469 ]
            wayne Wayne Siu made changes -
            Labels approved-for-7.0.1
            abhinav Abhinav Dangeti made changes -
            Fix Version/s Neo [ 17615 ]
            abhinav Abhinav Dangeti made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]

            Build couchbase-server-7.0.1-5960 contains cbgt commit e2b0115 with commit message:
            MB-47457: NetworkType:default for gocbcore Agent/DCPAgent configs

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.1-5960 contains cbgt commit e2b0115 with commit message: MB-47457 : NetworkType:default for gocbcore Agent/DCPAgent configs

            Build couchbase-server-7.1.0-1096 contains cbgt commit e2b0115 with commit message:
            MB-47457: NetworkType:default for gocbcore Agent/DCPAgent configs

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1096 contains cbgt commit e2b0115 with commit message: MB-47457 : NetworkType:default for gocbcore Agent/DCPAgent configs

            Verified in 7.0.1-5960 - following scenarios were tested on a 3 node CB cluster deployed on GKE with CAO 2.2.0:

            Create/Update/Drop FTS indexes
            Running queries from UI and REST
            Flex queries from N1QL (MB-47678)
            Drop Buckets should cover up basic scenarios

            Same scenarios will be executed on 7.1.0 as well

            CC Girish BenakappaMihir KamdarRoo ThorpRaju Suravarjjala

            arunkumar Arunkumar Senthilnathan added a comment - Verified in 7.0.1-5960 - following scenarios were tested on a 3 node CB cluster deployed on GKE with CAO 2.2.0: Create/Update/Drop FTS indexes Running queries from UI and REST Flex queries from N1QL ( MB-47678 ) Drop Buckets should cover up basic scenarios Same scenarios will be executed on 7.1.0 as well CC Girish Benakappa Mihir Kamdar Roo Thorp Raju Suravarjjala
            abhinav Abhinav Dangeti made changes -
            Link This issue relates to MB-47678 [ MB-47678 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.2 [ 18012 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            wayne Wayne Siu made changes -
            Labels approved-for-7.0.1 approved-for-7.0.1 approved-for-7.0.2
            wayne Wayne Siu made changes -
            Link This issue backports to MB-47678 [ MB-47678 ]
            wayne Wayne Siu made changes -
            Link This issue relates to MB-47678 [ MB-47678 ]

            Build couchbase-server-7.0.0-5306 contains cbgt commit 569ca44 with commit message:
            MB-47457: [BP] NetworkType:default for gocbcore Agent/DCPAgent configs

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-5306 contains cbgt commit 569ca44 with commit message: MB-47457 : [BP] NetworkType:default for gocbcore Agent/DCPAgent configs

            Build couchbase-server-7.0.1-6100 contains cbgt commit 5a75a31 with commit message:
            MB-47457: [BP] NetworkType:default for gocbcore Agent/DCPAgent configs

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.1-6100 contains cbgt commit 5a75a31 with commit message: MB-47457 : [BP] NetworkType:default for gocbcore Agent/DCPAgent configs

            Verified in 7.0.1 build 6102

            Create/Update/Drop FTS indexes
            Running queries from UI and REST
            Flex queries from N1QL
            Drop Buckets should cover up basic scenarios

            girish.benakappa Girish Benakappa added a comment - Verified in 7.0.1 build 6102 Create/Update/Drop FTS indexes Running queries from UI and REST Flex queries from N1QL Drop Buckets should cover up basic scenarios
            ritam.sharma Ritam Sharma made changes -
            Remote Link This issue links to "Page (Couchbase, Inc. Wiki)" [ 23031 ]
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Labels approved-for-7.0.1 approved-for-7.0.2 7.0.2-verified approved-for-7.0.1 approved-for-7.0.2
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Assignee Abhinav Dangeti [ abhinav ] Girish Benakappa [ girish.benakappa ]
            pvarley Patrick Varley made changes -
            Component/s tools [ 10223 ]
            pvarley Patrick Varley made changes -
            Component/s tools [ 10223 ]

            Verified with 7.1.0-2021. Closing the issue.

            girish.benakappa Girish Benakappa added a comment - Verified with 7.1.0-2021. Closing the issue.
            girish.benakappa Girish Benakappa made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              girish.benakappa Girish Benakappa
              matt.carabine Matt Carabine
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty