Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-1675

XDCR with Operator hits error when setting up with Buckets of the same name

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • None
    • 2.1.0
    • kubernetes, operator
    • None
    • 39: Bk Vol. Bugs. Ptwx.
    • 1

    Description

      Description:

      When creating XDCR with source and target buckets of the same name, only 1 direction would work, the other direction would hit the following error:

       

      {"level":"error","ts":1600123046.6096296,"logger":"cluster","msg":"Status update failed","cluster":"default/cb-test","error":"admission webhook \"couchbase-operator-admission.default.svc\" denied the request: validation failure list:\nbucket travel-sample referenced by spec.bucket in couchbasereplications.couchbase.com/xdcr-test-test must exist: bucket travel-sample not found","stacktrace":"github.com/couchbase/couchbase-operator/vendor/github.com/go-logr/zapr.(*zapLogger).
       
      

       

      Steps to re-create:

      1) Create 2 clusters in the same namespace, (cb-example.yaml and cb-test.yaml)

      2) Create 2 buckets with couchbase-bucket.yaml

       apiVersion: couchbase.com/v2 
      kind: CouchbaseBucket 
      metadata: 
        name: travel-test 
        labels: 
          cluster: cb-test 
      spec: 
        name: travel-sample 
        memoryQuota: 256Mi 
        replicas: 2 
        ioPriority: low 
        evictionPolicy: valueOnly 
        conflictResolution: lww 
        enableFlush: false 
        enableIndexReplica: true 
        compressionMode: passive 
      --- 
      apiVersion: couchbase.com/v2 
      kind: CouchbaseBucket 
      metadata: 
        name: travel-sample 
        labels: 
          cluster: cb-example 
      spec: 
        name: travel-sample 
        memoryQuota: 256Mi 
        replicas: 2 
        ioPriority: low 
        evictionPolicy: valueOnly 
        conflictResolution: lww 
        enableFlush: false 
        enableIndexReplica: true 
        compressionMode: passive 
      ---

       

      3) create XDCR from cb-example to cb-test with xdcrexample.yaml

      apiVersion: couchbase.com/v2
      kind: CouchbaseReplication
      metadata:
       name: xdcr-example-test-travel
       labels:
       cluster: cb-example
      spec:
       bucket: travel-sample
       remoteBucket: travel-sample
       paused: false

      This works without issue.

      4) Create XDCR from cb-test to cb-example with xdcrtest.yaml:

      apiVersion: couchbase.com/v2
      kind: CouchbaseReplication
      metadata:
       name: xdcr-test-sample-travel
       labels:
       cluster: cb-test
      spec:
       bucket: travel-sample
       remoteBucket: travel-sample
       paused: false

      5) The operator would show this error in the logs:

      {"level":"error","ts":1600123895.1002529,"logger":"cluster","msg":"Status update failed","cluster":"default/cb-test","error":"adm
      ission webhook \"couchbase-operator-admission.default.svc\" denied the request: validation failure list:\nbucket travel-sample re
      ferenced by spec.bucket in couchbasereplications.couchbase.com/xdcr-test-sample-travel must exist: bucket travel-sample not found
      ","stacktrace":"github.com/couchbase/couchbase-operator/vendor/github.com/go-logr/zapr.(*zapLogger).
      

       

       

      The error goes away if using with a different bucket name for source or remote.

      Attachments

        1. cb-example.yaml
          2 kB
        2. cbopinfo-20200915T133936-0700.tar.gz
          818 kB
        3. cb-test.yaml
          2 kB
        4. couchbase-bucket.yaml
          0.6 kB
        5. xdcrexample.yaml
          0.2 kB
        6. xdcrtest.yaml
          0.2 kB
        For Gerrit Dashboard: K8S-1675
        # Subject Branch Project Status CR V

        Activity

          simon.murray Simon Murray added a comment -

          This is working as designed.  The cluster cb-test is looking for bucket travel-sample, however the only bucket it can see is travel-test.  It must reference a unique Kubernetes resource name, that is selected by the cluster, not a meta name.

          simon.murray Simon Murray added a comment - This is working as designed.  The cluster cb-test is looking for bucket travel-sample , however the only bucket it can see is travel-test .  It must reference a unique Kubernetes resource name , that is selected by the cluster, not a meta name.
          tin.tran Tin Tran added a comment -

          Hi Simon Murray 

          Is there a requirement for both the metadata.name and the spec.name of the bucket to match in order for XDCR to work? Because I observed the following:

          1) If I create a bucket using different metadata.name and spec.name, like so

           

          apiVersion: couchbase.com/v2 
          kind: CouchbaseBucket 
          metadata: 
            name: bucket-test 
            labels: 
              cluster: cb-test 
          spec: 
            name: test 
            memoryQuota: 256Mi 
            replicas: 2 
            ioPriority: low 
            evictionPolicy: valueOnly 
            conflictResolution: lww 
            enableFlush: false 
            enableIndexReplica: true 
            compressionMode: passive 
          --- 
          apiVersion: couchbase.com/v2 
          kind: CouchbaseBucket 
          metadata: 
            name: bucket-sample 
            labels: 
              cluster: cb-example 
          spec: 
            name: test 
            memoryQuota: 256Mi 
            replicas: 2 
            ioPriority: low 
            evictionPolicy: valueOnly 
            conflictResolution: lww 
            enableFlush: false 
            enableIndexReplica: true 
            compressionMode: passive 
          ---

          Then XDCR would not work, when I reference the resource name, which is the metadata.name, I get the following error:

          apiVersion: couchbase.com/v2 
          kind: CouchbaseReplication 
          metadata: 
            name: xdcr-example-test-travel 
            labels: 
              cluster: cb-example 
          spec: 
            bucket: bucket-sample 
            remoteBucket: test 
            paused: false

           

          {"level":"error","ts":1600201367.3085656,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example","error":"[request
           failed POST http://cb-example-0000.cb-example.default.svc:8091/controller/createReplication 400 Bad Request: {\"errors\":{\"fromBucket\":
          \"Error validating source bucket 'bucket-sample'. err=BucketValidationInfo Operation failed after max retries.  Last error: Bucket doesn't
           exist\"}}], [request failed POST http://cb-example-0001.cb-example.default.svc:8091/controller/createReplication 400 Bad Request: {\"erro
          rs\":{\"fromBucket\":\"Error validating source bucket 'bucket-sample'. err=BucketValidationInfo Operation failed after max retries.  Last 
          error: Bucket doesn't exist\"}}]",

          If I reference the spec.name of the bucket, I get the following error:

          apiVersion: couchbase.com/v2 
          kind: CouchbaseReplication 
          metadata: 
            name: xdcr-example-test-travel 
            labels: 
              cluster: cb-example 
          spec: 
            bucket: test 
            remoteBucket: test 
            paused: false

          {"level":"error","ts":1600200827.6209834,"logger":"cluster","msg":"Cluster status update failed","cluster":"default/cb-example","error":"a
          dmission webhook \"couchbase-operator-admission.default.svc\" denied the request: validation failure list:\nbucket test referenced by spec
          .bucket in couchbasereplications.couchbase.com/xdcr-example-test-travel must exist: bucket test not found",

           

          Only when both metadata.name and spec.name the same that these errors would not appear. Moreover, if I use the metadata.name for the target bucket, I get the following error:

          apiVersion: couchbase.com/v2 
          kind: CouchbaseReplication 
          metadata: 
            name: xdcr-example-test-travel 
            labels: 
              cluster: cb-example 
          spec: 
            bucket: bucket-sample 
            remoteBucket: bucket-test
            paused: false

           

          "level":"error","ts":1600201558.1039696,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example","error":"[request
           failed POST http://cb-example-0001.cb-example.default.svc:8091/controller/createReplication 400 Bad Request: {\"errors\":{\"toBucket\":\"
          Error validating target bucket 'bucket-test'. err=BucketValidationInfo Operation failed after max retries.  Last error: Bucket doesn't exi
          st\"}}], [request failed POST http://cb-example-0000.cb-example.default.svc:8091/controller/createReplication 400 Bad Request: {\"errors\"
          :{\"toBucket\":\"Error validating target bucket 'bucket-test'. err=BucketValidationInfo Operation failed after max retries.  Last error: B
          ucket doesn't exist\"}}]",

           

          Only when I specified the replication with the spec.name for the remote bucket then XDCR was able to be created.

           

          apiVersion: couchbase.com/v2 
          kind: CouchbaseReplication 
          metadata: 
            name: xdcr-example-test-travel 
            labels: 
              cluster: cb-example 
          spec: 
            bucket: bucket-sample 
            remoteBucket: test 
            paused: false

           

          Attaching my cbopinfo logs here:

          cbopinfo-20200915T133936-0700.tar.gz

           

          Thank you Simon

           

          tin.tran Tin Tran added a comment - Hi Simon Murray   Is there a requirement for both the metadata.name and the spec.name of the bucket to match in order for XDCR to work? Because I observed the following: 1) If I create a bucket using different metadata.name and spec.name, like so   apiVersion: couchbase.com/v2 kind: CouchbaseBucket metadata:  name: bucket-test  labels:    cluster: cb-test spec:  name: test  memoryQuota: 256Mi  replicas: 2  ioPriority: low  evictionPolicy: valueOnly  conflictResolution: lww  enableFlush: false  enableIndexReplica: true  compressionMode: passive --- apiVersion: couchbase.com/v2 kind: CouchbaseBucket metadata:  name: bucket-sample  labels:    cluster: cb-example spec:  name: test  memoryQuota: 256Mi  replicas: 2  ioPriority: low  evictionPolicy: valueOnly  conflictResolution: lww  enableFlush: false  enableIndexReplica: true  compressionMode: passive --- Then XDCR would not work, when I reference the resource name, which is the metadata.name, I get the following error: apiVersion: couchbase.com/v2 kind: CouchbaseReplication metadata:  name: xdcr-example-test-travel  labels:    cluster: cb-example spec:  bucket: bucket-sample  remoteBucket: test  paused: false   {"level":"error","ts":1600201367.3085656,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example","error":"[request failed POST http://cb-example-0000.cb-example.default.svc:8091/controller/createReplication 400 Bad Request: {\"errors\":{\"fromBucket\": \"Error validating source bucket 'bucket-sample'. err=BucketValidationInfo Operation failed after max retries.  Last error: Bucket doesn't exist\"}}], [request failed POST http://cb-example-0001.cb-example.default.svc:8091/controller/createReplication 400 Bad Request: {\"erro rs\":{\"fromBucket\":\"Error validating source bucket 'bucket-sample'. err=BucketValidationInfo Operation failed after max retries.  Last error: Bucket doesn't exist\"}}]", If I reference the spec.name of the bucket, I get the following error: apiVersion: couchbase.com/v2 kind: CouchbaseReplication metadata:  name: xdcr-example-test-travel  labels:    cluster: cb-example spec:  bucket: test  remoteBucket: test  paused: false {"level":"error","ts":1600200827.6209834,"logger":"cluster","msg":"Cluster status update failed","cluster":"default/cb-example","error":"a dmission webhook \"couchbase-operator-admission.default.svc\" denied the request: validation failure list:\nbucket test referenced by spec .bucket in couchbasereplications.couchbase.com/xdcr-example-test-travel must exist: bucket test not found",   Only when both metadata.name and spec.name the same that these errors would not appear. Moreover, if I use the metadata.name for the target bucket, I get the following error: apiVersion: couchbase.com/v2 kind: CouchbaseReplication metadata:  name: xdcr-example-test-travel  labels:    cluster: cb-example spec:  bucket: bucket-sample   remoteBucket: bucket-test  paused: false   "level":"error","ts":1600201558.1039696,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example","error":"[request failed POST http://cb-example-0001.cb-example.default.svc:8091/controller/createReplication 400 Bad Request: {\"errors\":{\"toBucket\":\" Error validating target bucket 'bucket-test'. err=BucketValidationInfo Operation failed after max retries.  Last error: Bucket doesn't exi st\"}}], [request failed POST http://cb-example-0000.cb-example.default.svc:8091/controller/createReplication 400 Bad Request: {\"errors\" :{\"toBucket\":\"Error validating target bucket 'bucket-test'. err=BucketValidationInfo Operation failed after max retries.  Last error: B ucket doesn't exist\"}}]",   Only when I specified the replication with the spec.name for the remote bucket then XDCR was able to be created.   apiVersion: couchbase.com/v2 kind: CouchbaseReplication metadata:  name: xdcr-example-test-travel  labels:    cluster: cb-example spec:  bucket: bucket-sample  remoteBucket: test  paused: false   Attaching my cbopinfo logs here: cbopinfo-20200915T133936-0700.tar.gz   Thank you Simon  
          simon.murray Simon Murray added a comment -

          Er, balls, good point   XDCR needs to translate the bucket name if it's overridden... I wonder if this problem affects roles too..?

          simon.murray Simon Murray added a comment - Er, balls, good point   XDCR needs to translate the bucket name if it's overridden... I wonder if this problem affects roles too..?
          tin.tran Tin Tran added a comment - - edited

          Hi Simon Murray

          Do you mean the name for setting up Roles might also be affected?

          tin.tran Tin Tran added a comment - - edited Hi Simon Murray Do you mean the name for setting up Roles might also be affected?
          simon.murray Simon Murray added a comment -

          Well roles can be applied to named buckets... does this mean kubernetes name, or couchbase name??

          simon.murray Simon Murray added a comment - Well roles can be applied to named buckets... does this mean kubernetes name, or couchbase name??
          tin.tran Tin Tran added a comment -

          I see, I will look into that.

          tin.tran Tin Tran added a comment - I see, I will look into that.

          People

            simon.murray Simon Murray
            tin.tran Tin Tran
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty