Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47901

FTS not indexing documents when node-to-node encryption enabled

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.0.1-6003

      • Create a multi-node cluster with search on all the nodes
      • Enable node-to-node encryption like below:

        ./couchbase-cli setting-autofailover -c localhost:8091 -u Administrator -p password --enable-auto-failover=0
        SUCCESS: Auto-failover settings modified
        [root@localhost bin]# ./couchbase-cli node-to-node-encryption -c localhost:8091 -u Administrator -p password --enable
        Turned on encryption for node: http://172.23.121.46:8091
        Turned on encryption for node: http://172.23.121.52:8091
        Turned on encryption for node: http://172.23.121.58:8091
        Turned on encryption for node: http://172.23.121.63:8091
        SUCCESS: Switched node-to-node encryption on
        [root@localhost bin]# ./couchbase-cli setting-security -c localhost:8091 -u Administrator -p password --set --cluster-encryption-level all
        SUCCESS: Security settings updated
        [root@localhost bin]# ./couchbase-cli setting-autofailover -c localhost:8091 -u Administrator -p password --enable-auto-failover=1 --auto-failover-timeout=120 --max-failovers=1
        SUCCESS: Auto-failover settings modified
        

      • Load travel-sample
      • Create a fts dynamic index with 20 partitions and 1 replica

      FTS index created but not indexing documents

      In the logs we see errors like:

      2021-08-11T15:37:42.883-07:00 [INFO] (GOCBCORE) Creating new dcp agent: &{UserAgent:fts:travel-sample:0b3b2857370cdc9f798b0719b6f27bc1-46769e64 MemdAddrs:[] HTTPAddrs:[127.0.0.1:18091] UseTLS:true BucketName:travel-sample NetworkType:default Auth:0x1d87c50 TLSRootCAProvider:0xa34520 UseCompression:false DisableDecompression:false DisableJSONHello:false DisableXErrorHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s AgentPriority:1 UseExpiryOpcode:false UseStreamID:true UseOSOBackfill:false BackfillOrder:1 DCPBufferSize:20971520 DisableBufferAcknowledgement:false}
      2021-08-11T15:37:42.883-07:00 [INFO] (GOCBCORE) CCCPPOLL: No nodes available to poll, return upstream
      2021-08-11T15:37:42.887-07:00 [WARN] (GOCBCORE) Failed to connect to host. Get https://127.0.0.1:18091/pools/default/bs/travel-sample: x509: certificate is valid for 172.23.121.52, not 127.0.0.1 -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:615
      2021-08-11T15:37:42.894-07:00 [WARN] feed_dcp_gocbcore: CreateDcpAgent, err: Get https://127.0.0.1:18091/pools/default/bs/travel-sample: x509: certificate is valid for 172.23.121.52, not 127.0.0.1 (close DCPAgent: 0xc0003d6480) -- cbgt.setupGocbcoreDCPAgent() at feed_dcp_gocbcore.go:372
      2021-08-11T15:37:42.894-07:00 [INFO] (GOCBCORE) SDK Version: gocbcore/v9.1.4
      2021-08-11T15:37:42.894-07:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:travel-sample UserAgent:test_1deaf4e562021d4e_fd1a648f UseTLS:false NetworkType:default Auth:0x1d87c50 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}
      2021-08-11T15:37:42.895-07:00 [INFO] (GOCBCORE) CCCPPOLL: No nodes available to poll, return upstream
      2021-08-11T15:37:42.898-07:00 [INFO] (GOCBCORE) Found couchbase bucket and HTTP poller in use. Resetting pollers to start cccp.
      2021-08-11T15:37:42.958-07:00 [INFO] gocbcore_utils: CreateAgent succeeded (agent: 0xc00cb884e0, bucket: travel-sample, name: test_1deaf4e562021d4e_fd1a648f)
      2021-08-11T15:37:42.959-07:00 [INFO] feed_dcp_gocbcore: Closing Agent (0xc00cb884e0)
      2021-08-11T15:37:42.959-07:00 [INFO] (GOCBCORE) SDK Version: gocbcore/v9.1.4
      2021-08-11T15:37:42.959-07:00 [INFO] (GOCBCORE) Creating new dcp agent: &{UserAgent:fts:travel-sample:0b3b2857370cdc9f798b0719b6f27bc1-3a6d61e5 MemdAddrs:[] HTTPAddrs:[127.0.0.1:18091] UseTLS:true BucketName:travel-sample NetworkType:default Auth:0x1d87c50 TLSRootCAProvider:0xa34520 UseCompression:false DisableDecompression:false DisableJSONHello:false DisableXErrorHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s AgentPriority:1 UseExpiryOpcode:false UseStreamID:true UseOSOBackfill:false BackfillOrder:1 DCPBufferSize:20971520 DisableBufferAcknowledgement:false}
      2021-08-11T15:37:42.959-07:00 [INFO] (GOCBCORE) CCCPPOLL: No nodes available to poll, return upstream
      2021-08-11T15:37:42.963-07:00 [WARN] (GOCBCORE) Failed to connect to host. Get https://127.0.0.1:18091/pools/default/bs/travel-sample: x509: certificate is valid for 172.23.121.52, not 127.0.0.1 -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:615
      2021-08-11T15:37:42.971-07:00 [WARN] feed_dcp_gocbcore: CreateDcpAgent, err: Get https://127.0.0.1:18091/pools/default/bs/travel-sample: x509: certificate is valid for 172.23.121.52, not 127.0.0.1 (close DCPAgent: 0xc000321100) -- cbgt.setupGocbcoreDCPAgent() at feed_dcp_gocbcore.go:372
      2021-08-11T15:37:42.971-07:00 [INFO] (GOCBCORE) SDK Version: gocbcore/v9.1.4
      2021-08-11T15:37:42.971-07:00 [INFO] (GOCBCORE) Creating new agent: &{MemdAddrs:[] HTTPAddrs:[127.0.0.1:8091] BucketName:travel-sample UserAgent:test_1deaf4e562021d4e_55ba9666 UseTLS:false NetworkType:default Auth:0x1d87c50 TLSRootCAProvider:<nil> UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:1m0s KVConnectTimeout:7s KvPoolSize:0 MaxQueueSize:0 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}
      2021-08-11T15:37:42.971-07:00 [INFO] (GOCBCORE) CCCPPOLL: No nodes available to poll, return upstream
      

      Logs:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-08-11T224055-ns_1%40172.23.121.46.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-08-11T224055-ns_1%40172.23.121.52.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-08-11T224055-ns_1%40172.23.121.58.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-08-11T224055-ns_1%40172.23.121.63.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.1.0-1213 contains cbgt commit 087a56c with commit message:
            MB-47901: Do not use TLS to connect to localhost

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1213 contains cbgt commit 087a56c with commit message: MB-47901 : Do not use TLS to connect to localhost

            Build couchbase-server-7.1.0-1213 contains n1fty commit e47a0b0 with commit message:
            MB-47901: Update gocbcore version

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1213 contains n1fty commit e47a0b0 with commit message: MB-47901 : Update gocbcore version

            Build couchbase-server-7.1.0-1213 contains query commit a7026a4 with commit message:
            MB-47901: Update gocbcore version

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1213 contains query commit a7026a4 with commit message: MB-47901 : Update gocbcore version

            Verified with 7.0.2-6603

            girish.benakappa Girish Benakappa added a comment - Verified with 7.0.2-6603

            Verified for 7.1.0-1282

            evgeny.makarenko Evgeny Makarenko (Inactive) added a comment - Verified for 7.1.0-1282

            People

              evgeny.makarenko Evgeny Makarenko (Inactive)
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty