Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45853

[Windows] - Critical errors like "RbacConfigReloadTask(): An error occurred while loading RBAC configuration from [c:/Program Files/Couchbase/Server/var/lib/couchbase/config/memcached.rbac] - syntax error while parsing value" seen in memcached logs

    XMLWordPrintable

Details

    • Untriaged
    • Windows 64-bit
    • 1
    • No

    Description

      Script to Repro

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_drop_recreate_rebalance.CollectionsDropRecreateRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=6,nodes_failover=2,bucket_spec=multi_bucket.buckets_1000_collections,recovery_type=delta,GROUP=set1'
      

      Steps to Repro
      1. Create a 5 node cluster
      ----------------------------------------------------------------------

      Nodes Services Version CPU Status

      ----------------------------------------------------------------------

      172.23.104.247 kv 7.0.0-4985-enterprise 6.32833333333 Cluster node
      172.23.105.1 None     <--- IN —
      172.23.105.137 None     <--- IN —
      172.23.105.183 None     <--- IN —
      172.23.106.116 None     <--- IN —

      ----------------------------------------------------------------------

      2. Create buckets/scopes/collections/data
      2021-04-21 16:24:27,200 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
      ------------------------------------------------------------------------

      Bucket Type Replicas Durability TTL Items RAM Quota RAM Used Disk Used

      ------------------------------------------------------------------------

      bucket1 ephemeral 2 none 0 5000 5242880000 502953456 170
      default couchbase 2 none 0 5000 5242880000 501996792 366227683

      ------------------------------------------------------------------------

      3. Hard Failover 2 nodes

      2021-04-21 16:24:27,236 | test  | INFO    | MainThread | [collections_drop_recreate_rebalance:load_collections_with_failover:234] failing over nodes [ip:172.23.104.247 port:8091 ssh_username:Administrator, ip:172.23.105.1 port:8091 ssh_username:Administrator]
      

      They fail with following error

      2021-04-21 16:27:00,142 | test  | ERROR   | pool-11-thread-25 | [rest_client:_http_request:748] POST http://172.23.104.247:8091/controller/failOver body: otpNode=ns_1%40172.23.105.1&allowUnsafe=false headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: status: 500, content: Cannot safely perform a failover at the moment Cannot safely perform a failover at the moment auth: Administrator:password
      

      At this point we see the following error on 172.23.105.183

      Administrator@WIN-1T98IIFH727 /cygdrive/c/Program Files/Couchbase/Server/var/lib/couchbase/logs
      $ grep CRITICAL *
      memcached.log.000833.txt:2021-04-21T16:26:46.064372-07:00 CRITICAL 20572: RbacConfigReloadTask(): An error occurred while loading RBAC configuration from [c:/Program Files/Couchbase/Server/var/lib/couchbase/config/memcached.rbac] [ {"ip":"127.0.0.1","port":59145} - {"ip":"127.0.0.1","port":11209} (System, <ud>@ns_server</ud>) ]: Failed to parse RBAC database: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal
      grep: rebalance: Is a directory
      

      cbcollect_attached.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            An issue with the fix was identified and is in review: http://review.couchbase.org/c/ns_server/+/153510. I'll assign the ticket back to Bala once the change is merged.

            steve.watanabe Steve Watanabe added a comment - An issue with the fix was identified and is in review: http://review.couchbase.org/c/ns_server/+/153510 . I'll assign the ticket back to Bala once the change is merged.

            The fix was merged so assigning back to Bala.

            steve.watanabe Steve Watanabe added a comment - The fix was merged so assigning back to Bala.

            Build couchbase-server-7.0.0-5161 contains ns_server commit 49cf19b with commit message:
            MB-45853 Fix flushing of message

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-5161 contains ns_server commit 49cf19b with commit message: MB-45853 Fix flushing of message

            Have started a run on 7.0.0-5172-enterprise with all the tests that used to fail before. Will close this once all the tests pass. So far looks good.

            Logs : http://qa.sc.couchbase.com/job/temp_win_private_job_3/29/console

            Balakumaran.Gopal Balakumaran Gopal added a comment - Have started a run on 7.0.0-5172-enterprise with all the tests that used to fail before. Will close this once all the tests pass. So far looks good. Logs : http://qa.sc.couchbase.com/job/temp_win_private_job_3/29/console

            Validated this on 7.0.0-5172-enterprise. All the 61 tests related to the failure passed

            See consoleText_MB-45853.log for logs.
            Marking this closed.

            Balakumaran.Gopal Balakumaran Gopal added a comment - Validated this on 7.0.0-5172-enterprise. All the 61 tests related to the failure passed See consoleText_MB-45853.log for logs. Marking this closed.

            People

              Balakumaran.Gopal Balakumaran Gopal
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty