Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51763

[Windows] Unable to connect Local link after killing memcached on all nodes of cluster

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Not a Bug
    • 7.1.0
    • 7.1.0
    • analytics
    • 7.1.0-2549
    • Untriaged
    • Windows 64-bit
    • 1
    • Yes

    Description

      Steps- 

      1. Create a cluster as below -
      Node Services CPU_utilization Mem_total Mem_free Swap_mem_used Active / Replica Version
      172.23.136.103 cbas 16.8736145602 5.99 GiB 3.80 GiB 4.13 GiB / 7.62 GiB 0 / 0 7.1.0-2549-enterprise
      172.23.136.104 cbas 37.8916666667 5.99 GiB 3.80 GiB 4.13 GiB / 7.62 GiB 0 / 0 7.1.0-2549-enterprise
      172.23.136.101 cbas 5.83333333333 5.99 GiB 4.06 GiB 4.05 GiB / 7.62 GiB 0 / 0 7.1.0-2549-enterprise
      172.23.136.105 index, kv, n1ql 1.06833333333 5.99 GiB 4.31 GiB 1.96 GiB / 7.62 GiB 0 / 0 7.1.0-2549-enterprise
      1. Create CBAS entities as follows - 3 dataverses, 5 datasets per dataverse, 10 indexes.
      2. Wait for ingestion to complete
      3. Set analytics replica to 2 and rebalance so that replicas take effect.
      4. Disconnect all local links.
      5. Wait for data to be persisted.
      6. Kill memcached on all nodes of cluster.
      7. Start a running count queries on all the datasets.
      8. Start loading data in KV buckets.
      9. Connect all local links.

      One of the Local link is unable to connect post killing memcached on all cluster nodes.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          ritam.sharma Ritam Sharma added a comment -

          Last passed on RC3.

          ritam.sharma Ritam Sharma added a comment - Last passed on RC3.
          ritam.sharma Ritam Sharma added a comment - Changes from RC3 - RC4 http://changelog.build.couchbase.com/?product=couchbase-server&fromVersion=7.1.0&fromBuild=2534&toVersion=7.1.0&toBuild=2549&f_asterixdb=on&f_cbas-core=on&f_eventing=on&f_eventing-ee=on&f_indexing=on&f_kv_engine=on&f_ns_server=on&f_testrunner=on
          umang.agrawal Umang added a comment -

          Last passed on build 7.1.0-2534

          umang.agrawal Umang added a comment - Last passed on build 7.1.0-2534

          cc: Daniel Owen

          I don't see an (analytics) issue here.

          It took memcached a few (8) seconds to realize the bucket-0 creation after getting killed:

          2022-04-07T08:46:36.355590-07:00 INFO ---------- Opening logfile: 
          2022-04-07T08:46:36.357892-07:00 INFO Couchbase version 7.1.0-2549 starting.
          ...
          2022-04-07T08:46:44.357619-07:00 INFO 1084: Bucket [bucket-0] created successfully
          2022-04-07T08:46:44.357666-07:00 WARNING Slow runtime for CreateBucket task on thread unknown. runtime: 7625 ms
          2022-04-07T08:46:44.357921-07:00 WARNING 1084: Slow operation: {"bucket":"","cid":"127.0.0.1:51002/0","command":"CREATE_BUCKET","duration":"7626 ms","packet":{"bodylen":726,"cas":0,"datatype":"raw","extlen":0,"key":"<ud>bucket-0</ud>","keylen":8,"magic":"ClientRequest","opaque":0,"opcode":"CREATE_BUCKET","vbucket":0},"peer":{"ip":"127.0.0.1","port":51002},"response":"Success","trace":"request=683347059500:7625520 execute=683347059500:121 execute=690972561500:18","worker_tid":6248}
          

          During this time, analytics attempted to get info on the bucket via memcached, which failed with bucket not found:

          2022-04-07T08:46:39.188-07:00 DEBU CBAS.netty.DcpConnectHandler [nioEventLoopGroup-2-1] Failed in the SELECT step due to bucket bucket-0 not found (0x1: Key not found)
          

          As a result, analytics correctly truncates the datasets associated with the bucket. Should the connect link be tried again, after memcached realizes it exists, I would expect the connect link to succeed, and the datasets to be reingested.

          If this is not a satisfactory answer, please reassign to couchbase-bucket for further comment.

          michael.blow Michael Blow added a comment - cc: Daniel Owen I don't see an (analytics) issue here. It took memcached a few (8) seconds to realize the bucket-0 creation after getting killed: 2022-04-07T08:46:36.355590-07:00 INFO ---------- Opening logfile: 2022-04-07T08:46:36.357892-07:00 INFO Couchbase version 7.1.0-2549 starting. ... 2022-04-07T08:46:44.357619-07:00 INFO 1084: Bucket [bucket-0] created successfully 2022-04-07T08:46:44.357666-07:00 WARNING Slow runtime for CreateBucket task on thread unknown. runtime: 7625 ms 2022-04-07T08:46:44.357921-07:00 WARNING 1084: Slow operation: {"bucket":"","cid":"127.0.0.1:51002/0","command":"CREATE_BUCKET","duration":"7626 ms","packet":{"bodylen":726,"cas":0,"datatype":"raw","extlen":0,"key":"<ud>bucket-0</ud>","keylen":8,"magic":"ClientRequest","opaque":0,"opcode":"CREATE_BUCKET","vbucket":0},"peer":{"ip":"127.0.0.1","port":51002},"response":"Success","trace":"request=683347059500:7625520 execute=683347059500:121 execute=690972561500:18","worker_tid":6248} During this time, analytics attempted to get info on the bucket via memcached, which failed with bucket not found: 2022-04-07T08:46:39.188-07:00 DEBU CBAS.netty.DcpConnectHandler [nioEventLoopGroup-2-1] Failed in the SELECT step due to bucket bucket-0 not found (0x1: Key not found) As a result, analytics correctly truncates the datasets associated with the bucket. Should the connect link be tried again, after memcached realizes it exists, I would expect the connect link to succeed, and the datasets to be reingested. If this is not a satisfactory answer, please reassign to couchbase-bucket for further comment.
          umang.agrawal Umang added a comment -

          Retested the failing scenario with some sleep for nodes to recover after killing memcachced and the test worked, hence closing this issue.

          umang.agrawal Umang added a comment - Retested the failing scenario with some sleep for nodes to recover after killing memcachced and the test worked, hence closing this issue.

          People

            umang.agrawal Umang
            umang.agrawal Umang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty