Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41628

Investigate situation where lease start time estimate is in the future.

    XMLWordPrintable

Details

    • 1

    Description

      Using the knobs provided in ns_server to decrease the lease time:

      curl http://localhost:9000/diag/eval -u Administrator:asdasd -d 'ns_config:set({mb_master, heartbeat_interval}, 500).'
      curl http://localhost:9000/diag/eval -u Administrator:asdasd -d 'ns_config:set({mb_master, timeout_interval_count}, 6).'
      curl http://localhost:9000/diag/eval -u Administrator:asdasd -d 'ns_config:set({leader_lease_acquire_worker, lease_time}, 5000).'
      curl http://localhost:9000/diag/eval -u Administrator:asdasd -d 'ns_config:set({leader_lease_acquire_worker, lease_grace_time}, 2000).'
      curl http://localhost:9000/diag/eval -u Administrator:asdasd -d 'ns_config:set({leader_lease_acquire_worker, lease_renew_after}, 500).'
      

      I see periodic times when these are emitted

      info.log:[ns_server:warn,2020-09-23T11:33:23.645-07:00,n_0@127.0.0.1:<0.28104.0>:leader_lease_acquire_worker:get_prev_acquire_estimate:208]Lease period start time estimate is in the future. The time on the agent node 'n_2@127.0.0.1' must be flowing at a faster pace.
      info.log:[ns_server:warn,2020-09-23T11:33:24.148-07:00,n_0@127.0.0.1:<0.28104.0>:leader_lease_acquire_worker:get_prev_acquire_estimate:208]Lease period start time estimate is in the future. The time on the agent node 'n_2@127.0.0.1' must be flowing at a faster pace.
      info.log:[ns_server:warn,2020-09-23T11:33:25.164-07:00,n_0@127.0.0.1:<0.28104.0>:leader_lease_acquire_worker:get_prev_acquire_estimate:208]Lease period start time estimate is in the future. The time on the agent node 'n_2@127.0.0.1' must be flowing at a faster pace.
      info.log:[ns_server:warn,2020-09-23T11:33:25.662-07:00,n_0@127.0.0.1:<0.28104.0>:leader_lease_acquire_worker:get_prev_acquire_estimate:208]Lease period start time estimate is in the future. The time on the agent node 'n_2@127.0.0.1' must be flowing at a faster pace.
      info.log:[ns_server:warn,2020-09-23T11:33:26.164-07:00,n_0@127.0.0.1:<0.28104.0>:leader_lease_acquire_worker:get_prev_acquire_estimate:208]Lease period start time estimate is in the future. The time on the agent node 'n_2@127.0.0.1' must be flowing at a faster pace.
      info.log:[ns_server:warn,2020-09-23T11:33:26.665-07:00,n_0@127.0.0.1:<0.28104.0>:leader_lease_acquire_worker:get_prev_acquire_estimate:208]Lease period start time estimate is in the future. The time on the agent node 'n_2@127.0.0.1' must be flowing at a faster pace.
      

      It's possible this is related to running three nodes, two instances of cbc-pillowfight, along with activity (e.g. chrome) on my macbook.

      I've gathered cbcollect_info bundles from the nodes:

      https://s3.amazonaws.com/cb-engineering/stevewatanabe-fast-lease-expiration/collectinfo-2020-09-23T202035-n_0%40127.0.0.1.zip
      https://s3.amazonaws.com/cb-engineering/stevewatanabe-fast-lease-expiration/collectinfo-2020-09-23T202035-n_1%40127.0.0.1.zip
      https://s3.amazonaws.com/cb-engineering/stevewatanabe-fast-lease-expiration/collectinfo-2020-09-23T202035-n_2%40127.0.0.1.zip
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            dfinlay Dave Finlay
            steve.watanabe Steve Watanabe
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty