Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50295

TLS: Setting cluster encryption level to strict in mixed mode

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Major
    • None
    • 6.6.5
    • secondary-index
    • None
    • Untriaged
    • 1
    • Unknown

    Description

      Upgrade from 6.6.4-9961 to 6.6.5 build 10072

      • 4 node 6.6.4 cluster with n1: kv+n1ql, n2: kv+n1ql+index, n3: query+index+search, n4: n1ql+index+search
      • Load travel-sample
      • Create a partitioned index 

        CREATE INDEX `idx2` ON `travel-sample`(`name`) PARTITION BY hash((meta().`id`)) WITH { "nodes":[ "172.23.96.141:8091","172.23.97.211:8091","172.23.97.212:8091" ], "num_partition":3 }

      • Turn on node-node encryption and encryption level to control and make sure below query works 

        select meta().id from `travel-sample` where name = "40-Mile Air"

      • Failover n3 and upgrade to 6.6.5 and add back
      • Update the encryption level to strict on n3
      • Now the above query returns below error: 

        [
        		 
          {
        		 
            "code": 5000,
        		 
            "msg": " EOF from [172.23.97.212:9101 172.23.97.211:9101 172.23.96.141:9101] - cause:  EOF from [172.23.97.212:9101 172.23.97.211:9101 172.23.96.141:9101]",
        		 
            "query": "select meta().id from `travel-sample` where name = \"40-Mile Air\""
        		 
          }
        		 
        ]

      From the logs of 172.23.97.211:

      2022-01-07T11:58:23.911-08:00 [Info] [Queryport-connpool:172.23.96.141:9101] open new connection ...
      		 
      2022-01-07T11:58:23.912-08:00 [Error] [GsiScanClient:"172.23.96.141:9101"] req(0d4e8475-41d7-4c03-a439-6d2783993466) connection "172.23.97.211:37922" closed `EOF` 
      2022-01-07T11:58:23.912-08:00 [Error] [GsiScanClient:"172.23.96.141:9101"] Scan3(0d4e8475-41d7-4c03-a439-6d2783993466) response failed `EOF`
      		 
      2022-01-07T11:58:23.912-08:00 [Warn] scan failed: requestId 0d4e8475-41d7-4c03-a439-6d2783993466 queryport 172.23.97.212:9101 inst 17171700956637086408 partition [1]
      		 
      2022-01-07T11:58:23.912-08:00 [Warn] scan failed: requestId 0d4e8475-41d7-4c03-a439-6d2783993466 queryport 172.23.97.211:9101 inst 17171700956637086408 partition [2]
      		 
      2022-01-07T11:58:23.912-08:00 [Warn] scan failed: requestId 0d4e8475-41d7-4c03-a439-6d2783993466 queryport 172.23.96.141:9101 inst 17171700956637086408 partition [3]
      		 
      2022-01-07T11:58:23.912-08:00 [Warn] Scan failed with error for index 1657320327829898322.  Trying scan again with replica, reqId:0d4e8475-41d7-4c03-a439-6d2783993466 :  EOF from [172.23.97.212:9101 172.23.97.211:9101 172.23.96.141:9101] ...
      		 
      2022-01-07T11:58:23.912-08:00 [Error] PickRandom: Fail to find indexer for all index partitions. Num partition 3.  Partition with instances 2 
      2022-01-07T11:58:23.912-08:00 [Warn] Fail to find indexers to satisfy query request.  Trying scan again for index 1657320327829898322, reqId:0d4e8475-41d7-4c03-a439-6d2783993466 :  EOF from [172.23.97.212:9101 172.23.97.211:9101 172.23.96.141:9101] ...
      		 
      2022-01-07T11:58:23.912-08:00 [Info] [Queryport-connpool:172.23.96.141:9101] closing unhealthy connection "172.23.97.211:37922"
      		 
      2022-01-07T11:58:23.924-08:00 [Info] [Queryport-connpool:172.23.96.141:9101] open new connection ...
      		 
      2022-01-07T11:58:23.924-08:00 [Error] [GsiScanClient:"172.23.96.141:9101"] req(0d4e8475-41d7-4c03-a439-6d2783993466) connection "172.23.97.211:37924" closed `EOF` 
      2022-01-07T11:58:23.925-08:00 [Info] [Queryport-connpool:172.23.96.141:9101] open new connection ...
      		 
      2022-01-07T11:58:23.925-08:00 [Info] [Queryport-connpool:172.23.96.141:9101] closing unhealthy connection "172.23.97.211:37924"
      		 
      2022-01-07T11:58:23.925-08:00 [Error] [GsiScanClient:"172.23.96.141:9101"] req(0d4e8475-41d7-4c03-a439-6d2783993466) connection "172.23.97.211:37926" closed `EOF` 
      2022-01-07T11:58:23.925-08:00 [Error] [GsiScanClient:"172.23.96.141:9101"] Scan3(0d4e8475-41d7-4c03-a439-6d2783993466) response failed `EOF`
      		 
      2022-01-07T11:58:23.925-08:00 [Warn] scan failed: requestId 0d4e8475-41d7-4c03-a439-6d2783993466 queryport 172.23.97.212:9101 inst 17171700956637086408 partition [1]
      		 
      2022-01-07T11:58:23.925-08:00 [Warn] scan failed: requestId 0d4e8475-41d7-4c03-a439-6d2783993466 queryport 172.23.97.211:9101 inst 17171700956637086408 partition [2]
      		 
      2022-01-07T11:58:23.925-08:00 [Warn] scan failed: requestId 0d4e8475-41d7-4c03-a439-6d2783993466 queryport 172.23.96.141:9101 inst 17171700956637086408 partition [3]
      		 
      2022-01-07T11:58:23.925-08:00 [Warn] Scan failed with error for index 1657320327829898322.  Trying scan again with replica, reqId:0d4e8475-41d7-4c03-a439-6d2783993466 :  EOF from [172.23.97.212:9101 172.23.97.211:9101 172.23.96.141:9101] ...
      		 
      2022-01-07T11:58:23.925-08:00 [Error] PickRandom: Fail to find indexer for all index partitions. Num partition 3.  Partition with instances 2 
      2022-01-07T11:58:23.925-08:00 [Warn] Fail to find indexers to satisfy query request.  Terminate scan for index 1657320327829898322,  reqId:0d4e8475-41d7-4c03-a439-6d2783993466 :  EOF from [172.23.97.212:9101 172.23.97.211:9101 172.23.96.141:9101]
      		 
      2022-01-07T11:58:23.925-08:00 [Info] [Queryport-connpool:172.23.96.141:9101] closing unhealthy connection "172.23.97.211:37926"

      Logs:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-01-07T201111-ns_1%40172.23.120.174.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-01-07T201111-ns_1%40172.23.96.141.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-01-07T201111-ns_1%40172.23.97.211.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-01-07T201111-ns_1%40172.23.97.212.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            girish.benakappa Girish Benakappa
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty