Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37290

[Durability-Upgrade]: Durability fails with FeatureNotAvailable even after all kv nodes are upgraded to MH version

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • Unknown

    Description

       

      Initial build: 5.5.4-4338

      MH Build: 6.5.0-3939

       

      Scenario:

      • Initiate 3 node cluster (2 nodes only KV & 3rd node with index,query) using build 5.5.4-4338
      • Create Couchbase bucket with replica=1
      • Upgrade only the KV nodes to MH build (6.5.0-4947)
      • Try sync_write to the already created bucket

      Observation:

      Sync_write fails with reason com.couchbase.client.core.error.FeatureNotAvailableException.

      Expected behavior:

      Since durability depends only on KV nodes, syncWrite should succeed with all KV nodes are upgraded to MH build.

       

      Cluster status:
      +----------------+-------------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+
      | Node           | Services    | CPU_utilization | Mem_total  | Mem_free   | Swap_mem_used         | Active / Replica  | Version               |
      +----------------+-------------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+
      | 10.112.191.103 | index, n1ql | 0               | 1930829824 | 1190797312 | 0 / 1107292160        | 0 / 0             | 5.5.4-4338-enterprise |
      | 10.112.191.101 | kv          | 4.16666666667   | 1930829824 | 1431461888 | 38539264 / 1107292160 | 0 / 0             | 6.5.0-3939-enterprise |
      | 10.112.191.104 | kv          | 37.2340425532   | 1930829824 | 1298415616 | 0 / 1107292160        | 0 / 0             | 6.5.0-3939-enterprise |
      +----------------+-------------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+2019-12-17 Bucket status:
      +---------+---------+----------+-----+-------+-----------+----------+-----------+
      | Bucket  | Type    | Replicas | TTL | Items | RAM Quota | RAM Used | Disk Used |
      +---------+---------+----------+-----+-------+-----------+----------+-----------+
      | default | membase | 1        | 0   | 0     | 209715200 | 95524408 | 27371646  |
      +---------+---------+----------+-----+-------+-----------+----------+-----------+2019-12-17 Doc loading result:
      +---------------+--------------------------------------------------------------------------------------------------------------------------------+
      | Create doc_Id | Exception                                                                                                                      |
      +---------------+--------------------------------------------------------------------------------------------------------------------------------+
      | test_docs-0   | com.couchbase.client.core.error.FeatureNotAvailableException: Synchronous Durability is currently not available on this bucket |
      +---------------+--------------------------------------------------------------------------------------------------------------------------------+
      

      Note: Behavior is same even while using latest MH builds.

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          ashwin.govindarajulu Ashwin Govindarajulu created issue -
          ritam.sharma Ritam Sharma made changes -
          Field Original Value New Value
          Environment 6.5.0-3939 6.5.0-4947
          ritam.sharma Ritam Sharma made changes -
          Description  

          *Initial build:* 5.5.4-4338

          *MH Build:* 6.5.0-3939

           

          *Scenario:*
           * Initiate 3 node cluster (2 nodes only KV & 3rd node with index,query) using build 5.5.4-4338
           * Create Couchbase bucket with replica=1
           * Upgrade only the KV nodes to MH build (6.5.0-3939)
           * Try sync_write to the already created bucket

          *Observation:*

          Sync_write fails with reason com.couchbase.client.core.error.FeatureNotAvailableException.

          *Expected behavior:*

          Since durability depends only on KV nodes, syncWrite should succeed with all KV nodes are upgraded to MH build.

           
          {noformat}
          Cluster status:
          +----------------+-------------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+
          | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
          +----------------+-------------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+
          | 10.112.191.103 | index, n1ql | 0 | 1930829824 | 1190797312 | 0 / 1107292160 | 0 / 0 | 5.5.4-4338-enterprise |
          | 10.112.191.101 | kv | 4.16666666667 | 1930829824 | 1431461888 | 38539264 / 1107292160 | 0 / 0 | 6.5.0-3939-enterprise |
          | 10.112.191.104 | kv | 37.2340425532 | 1930829824 | 1298415616 | 0 / 1107292160 | 0 / 0 | 6.5.0-3939-enterprise |
          +----------------+-------------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+2019-12-17 Bucket status:
          +---------+---------+----------+-----+-------+-----------+----------+-----------+
          | Bucket | Type | Replicas | TTL | Items | RAM Quota | RAM Used | Disk Used |
          +---------+---------+----------+-----+-------+-----------+----------+-----------+
          | default | membase | 1 | 0 | 0 | 209715200 | 95524408 | 27371646 |
          +---------+---------+----------+-----+-------+-----------+----------+-----------+2019-12-17 Doc loading result:
          +---------------+--------------------------------------------------------------------------------------------------------------------------------+
          | Create doc_Id | Exception |
          +---------------+--------------------------------------------------------------------------------------------------------------------------------+
          | test_docs-0 | com.couchbase.client.core.error.FeatureNotAvailableException: Synchronous Durability is currently not available on this bucket |
          +---------------+--------------------------------------------------------------------------------------------------------------------------------+
          {noformat}
          *Note*: Behavior is same even while using latest MH builds.

           
           

          *Initial build:* 5.5.4-4338

          *MH Build:* 6.5.0-3939

           

          *Scenario:*
           * Initiate 3 node cluster (2 nodes only KV & 3rd node with index,query) using build 5.5.4-4338
           * Create Couchbase bucket with replica=1
           * Upgrade only the KV nodes to MH build (6.5.0-4947)
           * Try sync_write to the already created bucket

          *Observation:*

          Sync_write fails with reason com.couchbase.client.core.error.FeatureNotAvailableException.

          *Expected behavior:*

          Since durability depends only on KV nodes, syncWrite should succeed with all KV nodes are upgraded to MH build.

           
          {noformat}
          Cluster status:
          +----------------+-------------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+
          | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
          +----------------+-------------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+
          | 10.112.191.103 | index, n1ql | 0 | 1930829824 | 1190797312 | 0 / 1107292160 | 0 / 0 | 5.5.4-4338-enterprise |
          | 10.112.191.101 | kv | 4.16666666667 | 1930829824 | 1431461888 | 38539264 / 1107292160 | 0 / 0 | 6.5.0-3939-enterprise |
          | 10.112.191.104 | kv | 37.2340425532 | 1930829824 | 1298415616 | 0 / 1107292160 | 0 / 0 | 6.5.0-3939-enterprise |
          +----------------+-------------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+2019-12-17 Bucket status:
          +---------+---------+----------+-----+-------+-----------+----------+-----------+
          | Bucket | Type | Replicas | TTL | Items | RAM Quota | RAM Used | Disk Used |
          +---------+---------+----------+-----+-------+-----------+----------+-----------+
          | default | membase | 1 | 0 | 0 | 209715200 | 95524408 | 27371646 |
          +---------+---------+----------+-----+-------+-----------+----------+-----------+2019-12-17 Doc loading result:
          +---------------+--------------------------------------------------------------------------------------------------------------------------------+
          | Create doc_Id | Exception |
          +---------------+--------------------------------------------------------------------------------------------------------------------------------+
          | test_docs-0 | com.couchbase.client.core.error.FeatureNotAvailableException: Synchronous Durability is currently not available on this bucket |
          +---------------+--------------------------------------------------------------------------------------------------------------------------------+
          {noformat}
          *Note*: Behavior is same even while using latest MH builds.

           
          drigby Dave Rigby added a comment -

          The Expected behavour is incorrect.

          There's only a single, cluster-wide cluster config managed by ns_server. As such, until all nodes in a cluster are upgraded to the new version, the cluster config doesn't advance (and "unlock" any new features such as SyncWrites).

          Just because all KV nodes have been upgraded doesn't mean a feature can necessarily be enabled - why this could work for Sync Writes, that's not the case for other features which may well require all nodes to be upgraded, particulary the ns_server on each node.

          drigby Dave Rigby added a comment - The Expected behavour is incorrect. There's only a single, cluster-wide cluster config managed by ns_server. As such, until all nodes in a cluster are upgraded to the new version, the cluster config doesn't advance (and "unlock" any new features such as SyncWrites). Just because all KV nodes have been upgraded doesn't mean a feature can necessarily be enabled - why this could work for Sync Writes, that's not the case for other features which may well require all nodes to be upgraded, particulary the ns_server on each node.
          drigby Dave Rigby made changes -
          Fix Version/s Mad-Hatter [ 15037 ]
          Assignee Daniel Owen [ owend ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
          Resolution Not a Bug [ 10200 ]
          Status Open [ 1 ] Resolved [ 5 ]

          Closing the ticket, since the behavior observed is valid.

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - Closing the ticket, since the behavior observed is valid.
          ashwin.govindarajulu Ashwin Govindarajulu made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty