Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47901

FTS not indexing documents when node-to-node encryption enabled

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.0.1-6003

      • Create a multi-node cluster with search on all the nodes
      • Enable node-to-node encryption like below: 

        ./couchbase-cli setting-autofailover -c localhost:8091 -u Administrator -p password --enable-auto-failover=0
        		 
        SUCCESS: Auto-failover settings modified
        		 
        [root@localhost bin]# ./couchbase-cli node-to-node-encryption -c localhost:8091 -u Administrator -p password --enable
        		 
        Turned on encryption for node: http://172.23.121.46:8091
        		 
        Turned on encryption for node: http://172.23.121.52:8091
        		 
        Turned on encryption for node: http://172.23.121.58:8091
        		 
        Turned on encryption for node: http://172.23.121.63:8091
        		 
        SUCCESS: Switched node-to-node encryption on
        		 
        [root@localhost bin]# ./couchbase-cli setting-security -c localhost:8091 -u Administrator -p password --set --cluster-encryption-level all
        		 
        SUCCESS: Security settings updated
        		 
        [root@localhost bin]# ./couchbase-cli setting-autofailover -c localhost:8091 -u Administrator -p password --enable-auto-failover=1 --auto-failover-timeout=120 --max-failovers=1
        		 
        SUCCESS: Auto-failover settings modified

      • Load travel-sample
      • Create a fts dynamic index with 20 partitions and 1 replica

      FTS index created but not indexing documents

      In the logs we see errors like:

      2021-08-11T15:37:42.883-07:00 [INFO] (GOCBCORE) Creating new dcp agent: &{UserAgent:fts:travel-sample:0b3b2857370cdc9f798b0719b6f27bc1-46769e64 MemdAddrs:[] HTTPAddrs:[127.0.0.1:18091] UseTLS:true BucketName:travel-sample NetworkType:default Auth:0x1d87c50 TLSRootCAProvider:0xa34520 UseCompression:false DisableDecompression:false DisableJSONHello:false DisableXErrorHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s AgentPriority:1 UseExpiryOpcode:false UseStreamID:true UseOSOBackfill:false BackfillOrder:1 DCPBufferSize:20971520 DisableBufferAcknowledgement:false}
      		 
      2021-08-11T15:37:42.883-07:00 [INFO] (GOCBCORE) CCCPPOLL: No nodes available to poll, return upstream
      		 
      2021-08-11T15:37:42.887-07:00 [WARN] (GOCBCORE) Failed to connect to host. Get https://127.0.0.1:18091/pools/default/bs/travel-sample: x509: certificate is valid for 172.23.121.52, not 127.0.0.1 -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:615
      		 
      2021-08-11T15:37:42.894-07:00 [WARN] feed_dcp_gocbcore: CreateDcpAgent, err: Get https://127.0.0.1:18091/pools/default/bs/travel-sample: x509: certificate is valid for 172.23.121.52, not 127.0.0.1 (close DCPAgent: 0xc0003d6480) -- cbgt.setupGocbcoreDCPAgent() at feed_dcp_gocbcore.go:372
      		 
      2021-08-11T15:37:42.894-07:00 [INFO] (GOCBCORE) SDK Version: gocbcore/v9.1.4
      		 
      2021-08-11T15:37:42.894-07:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:travel-sample UserAgent:test_1deaf4e562021d4e_fd1a648f UseTLS:false NetworkType:default Auth:0x1d87c50 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}
      		 
      2021-08-11T15:37:42.895-07:00 [INFO] (GOCBCORE) CCCPPOLL: No nodes available to poll, return upstream
      		 
      2021-08-11T15:37:42.898-07:00 [INFO] (GOCBCORE) Found couchbase bucket and HTTP poller in use. Resetting pollers to start cccp.
      		 
      2021-08-11T15:37:42.958-07:00 [INFO] gocbcore_utils: CreateAgent succeeded (agent: 0xc00cb884e0, bucket: travel-sample, name: test_1deaf4e562021d4e_fd1a648f)
      		 
      2021-08-11T15:37:42.959-07:00 [INFO] feed_dcp_gocbcore: Closing Agent (0xc00cb884e0)
      		 
      2021-08-11T15:37:42.959-07:00 [INFO] (GOCBCORE) SDK Version: gocbcore/v9.1.4
      		 
      2021-08-11T15:37:42.959-07:00 [INFO] (GOCBCORE) Creating new dcp agent: &{UserAgent:fts:travel-sample:0b3b2857370cdc9f798b0719b6f27bc1-3a6d61e5 MemdAddrs:[] HTTPAddrs:[127.0.0.1:18091] UseTLS:true BucketName:travel-sample NetworkType:default Auth:0x1d87c50 TLSRootCAProvider:0xa34520 UseCompression:false DisableDecompression:false DisableJSONHello:false DisableXErrorHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s AgentPriority:1 UseExpiryOpcode:false UseStreamID:true UseOSOBackfill:false BackfillOrder:1 DCPBufferSize:20971520 DisableBufferAcknowledgement:false}
      		 
      2021-08-11T15:37:42.959-07:00 [INFO] (GOCBCORE) CCCPPOLL: No nodes available to poll, return upstream
      		 
      2021-08-11T15:37:42.963-07:00 [WARN] (GOCBCORE) Failed to connect to host. Get https://127.0.0.1:18091/pools/default/bs/travel-sample: x509: certificate is valid for 172.23.121.52, not 127.0.0.1 -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:615
      		 
      2021-08-11T15:37:42.971-07:00 [WARN] feed_dcp_gocbcore: CreateDcpAgent, err: Get https://127.0.0.1:18091/pools/default/bs/travel-sample: x509: certificate is valid for 172.23.121.52, not 127.0.0.1 (close DCPAgent: 0xc000321100) -- cbgt.setupGocbcoreDCPAgent() at feed_dcp_gocbcore.go:372
      		 
      2021-08-11T15:37:42.971-07:00 [INFO] (GOCBCORE) SDK Version: gocbcore/v9.1.4
      		 
      2021-08-11T15:37:42.971-07:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:travel-sample UserAgent:test_1deaf4e562021d4e_55ba9666 UseTLS:false NetworkType:default Auth:0x1d87c50 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}
      		 
      2021-08-11T15:37:42.971-07:00 [INFO] (GOCBCORE) CCCPPOLL: No nodes available to poll, return upstream

      Logs:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-08-11T224055-ns_1%40172.23.121.46.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-08-11T224055-ns_1%40172.23.121.52.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-08-11T224055-ns_1%40172.23.121.58.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-08-11T224055-ns_1%40172.23.121.63.zip

      Attachments

        Issue Links

          depends on

          Improvement - An improvement or enhancement to an existing feature or task. GOCBC-1162 Initial bootstrap of SDK should allow nonTLS connection when UseTLS is true

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-48148 [UI][Enforce TLS: Strict] No Docs created for FTS_Index

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-48780 FTS does not work if N2n encryption is set to 'all'

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-47982 FTS rebalance exited with reason ; service_rebalance_failed , agent_died

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-48111 enforce TLS: pindex partitions not created for empty bucket

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-42364 Enforce TLS - FTS

          • Major - Major loss of function.
          • Closed

          Activity

            People

              evgeny.makarenko Evgeny Makarenko (Inactive)
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty