Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39615

[BucketDurability]: Updating bucket_level_durability with diag_eval turns bucket into stale state

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: User Error
    • Affects Version/s: 6.6.0
    • Fix Version/s: None
    • Component/s: test-execution
    • Environment:
      Enterprise Edition 6.6.0 build 7687
      Centos7 on vagrant
    • Triage:
      Untriaged
    • Operating System:
      Centos 64-bit
    • Story Points:
      1
    • Is this a Regression?:
      Unknown

      Description

      Build: 6.6.0 - 7687 **

      Scenario:

      1. Two node cluster, turn off auto_failover
      2. Create Couchbase bucket(replica=1, bucket_durability=MAJORITY)
      3. Load 10K docs into the bucket
      4. Update bucket_durability=majorityAndPersistActive (ns_bucket:update_bucket_props)
      5. Restart kv_servers after diag_eval update (/bin/systemctl restart couchbase-server.service)

      Observation:

      • After restarting servers, bucket turns into "amber" color stale state.
      • Also seeing compactor error from UI logs continuously.
      • Bucket_min_durability has not been updated.

      Cb server logs:
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/bucket_d_update_issue/collectinfo-2020-05-28T115720-ns_1%4010.112.191.101.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/bucket_d_update_issue/collectinfo-2020-05-28T115720-ns_1%4010.112.191.102.zip

        Attachments

        1. bucket_state.png
          bucket_state.png
          181 kB
        2. server_state.png
          server_state.png
          528 kB
        3. test.log
          7 kB
        4. UI_logs.png
          UI_logs.png
          465 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          drigby Dave Rigby added a comment -

          Ashwin Govindarajulu Can you add the exact diag/eval you used to modify bucket_durability?

          Note that isn't a supported interface, so I'm not sure this necessary a valid bug.

          Show
          drigby Dave Rigby added a comment - Ashwin Govindarajulu Can you add the exact diag/eval you used to modify bucket_durability? Note that isn't a supported interface, so I'm not sure this necessary a valid bug.
          Hide
          ashwin.govindarajulu Ashwin Govindarajulu added a comment -

          Dave Rigby- Please find the details about our diag_eval call in framework.

          As part of scenario #4, we run the following commands on each kv-node.

          Run on each kv-node directly:
          curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
           
          This is called directly from external CURL call:
          curl http://10.112.191.101:8091/diag/eval -X POST -u Administrator -p password -d 'ns_bucket:update_bucket_props("default", [{extra_config_string, "durability_min_level=majorityAndPersistActive"}]).'

           
          Show
          ashwin.govindarajulu Ashwin Govindarajulu added a comment - Dave Rigby - Please find the details about our diag_eval call in framework. As part of scenario #4, we run the following commands on each kv-node. Run on each kv-node directly: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'   This is called directly from external CURL call: curl http://10.112.191.101:8091/diag/eval -X POST -u Administrator -p password -d 'ns_bucket:update_bucket_props("default", [{extra_config_string, "durability_min_level=majorityAndPersistActive"}]).'  
          Hide
          drigby Dave Rigby added a comment -

          From the memcached.log:

          2020-05-28T04:58:13.588881-07:00 WARNING 43 - Failed to create bucket [default]: Validation Error, durability_min_level takes one of [majority, majority_and_persist_on_master, none, persist_to_majority] (Got: majorityAndPersistActive)
          

          Show
          drigby Dave Rigby added a comment - From the memcached.log: 2020-05-28T04:58:13.588881-07:00 WARNING 43 - Failed to create bucket [default]: Validation Error, durability_min_level takes one of [majority, majority_and_persist_on_master, none, persist_to_majority] (Got: majorityAndPersistActive)
          Hide
          ashwin.govindarajulu Ashwin Govindarajulu added a comment -

          Closing this ticket.

          Not seeing this issue while using correct bucket_durability_values in diag_eval update.

          Show
          ashwin.govindarajulu Ashwin Govindarajulu added a comment - Closing this ticket. Not seeing this issue while using correct bucket_durability_values in diag_eval update.

            People

            Assignee:
            ashwin.govindarajulu Ashwin Govindarajulu
            Reporter:
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty