Details
Major Description
Steps-
- Create a cluster as below -
| Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
| 172.23.136.103 | cbas | 16.8736145602 | 5.99 GiB | 3.80 GiB | 4.13 GiB / 7.62 GiB | 0 / 0 | 7.1.0-2549-enterprise |
| 172.23.136.104 | cbas | 37.8916666667 | 5.99 GiB | 3.80 GiB | 4.13 GiB / 7.62 GiB | 0 / 0 | 7.1.0-2549-enterprise |
| 172.23.136.101 | cbas | 5.83333333333 | 5.99 GiB | 4.06 GiB | 4.05 GiB / 7.62 GiB | 0 / 0 | 7.1.0-2549-enterprise |
| 172.23.136.105 | index, kv, n1ql | 1.06833333333 | 5.99 GiB | 4.31 GiB | 1.96 GiB / 7.62 GiB | 0 / 0 | 7.1.0-2549-enterprise |
- Create CBAS entities as follows - 3 dataverses, 5 datasets per dataverse, 10 indexes.
- Wait for ingestion to complete
- Set analytics replica to 2 and rebalance so that replicas take effect.
- Disconnect all local links.
- Wait for data to be persisted.
- Kill memcached on all nodes of cluster.
- Start a running count
queries on all the datasets. - Start loading data in KV buckets.
- Connect all local links.
One of the Local link is unable to connect post killing memcached on all cluster nodes.
Attachments
Activity
| Field | Original Value | New Value |
|---|---|---|
| Fix Version/s | Neo [ 17615 ] | |
| Is this a Regression? | Unknown [ 10452 ] | Yes [ 10450 ] |
| Operating System | Windows 64-bit [ 10024 ] | |
| Description |
Steps-
# Create a cluster as below - |Node|Services|CPU_utilization|Mem_total|Mem_free|Swap_mem_used|Active / Replica|Version| |172.23.136.103|cbas|16.8736145602|5.99 GiB|3.80 GiB|4.13 GiB / 7.62 GiB|0 / 0|7.1.0-2549-enterprise| |172.23.136.104|cbas|37.8916666667|5.99 GiB|3.80 GiB|4.13 GiB / 7.62 GiB|0 / 0|7.1.0-2549-enterprise| |172.23.136.101|cbas|5.83333333333|5.99 GiB|4.06 GiB|4.05 GiB / 7.62 GiB|0 / 0|7.1.0-2549-enterprise| |172.23.136.105|index, kv, n1ql|1.06833333333|5.99 GiB|4.31 GiB|1.96 GiB / 7.62 GiB|0 / 0|7.1.0-2549-enterprise| # Create CBAS entities as follows - 3 dataverses, 5 datasets per dataverse, 10 indexes. # Wait for ingestion to complete # Set analytics replica to 2 and rebalance so that replicas take effect. # Disconnect all local links. # Wait for data to be persisted. # Kill memcached on all nodes of cluster. # Start a running count(*) queries on all the datasets. # Start loading data in KV buckets. # Connect all local links. One of the Local link is unable to connect post killing memcached on all cluster nodes. |
|
| Environment | 7.1.0-2549 | |
| Labels | analytics windows |
| Attachment | collectinfo-2022-04-07T154649-ns_1@172.23.136.101.zip [ 181512 ] | |
| Attachment | collectinfo-2022-04-07T154649-ns_1@172.23.136.103.zip [ 181513 ] | |
| Attachment | collectinfo-2022-04-07T154649-ns_1@172.23.136.104.zip [ 181514 ] | |
| Attachment | collectinfo-2022-04-07T154649-ns_1@172.23.136.105.zip [ 181515 ] |
cc: Daniel Owen
I don't see an (analytics) issue here.
It took memcached a few (8) seconds to realize the bucket-0 creation after getting killed:
2022-04-07T08:46:36.355590-07:00 INFO ---------- Opening logfile:
|
2022-04-07T08:46:36.357892-07:00 INFO Couchbase version 7.1.0-2549 starting.
|
...
|
2022-04-07T08:46:44.357619-07:00 INFO 1084: Bucket [bucket-0] created successfully
|
2022-04-07T08:46:44.357666-07:00 WARNING Slow runtime for CreateBucket task on thread unknown. runtime: 7625 ms
|
2022-04-07T08:46:44.357921-07:00 WARNING 1084: Slow operation: {"bucket":"","cid":"127.0.0.1:51002/0","command":"CREATE_BUCKET","duration":"7626 ms","packet":{"bodylen":726,"cas":0,"datatype":"raw","extlen":0,"key":"<ud>bucket-0</ud>","keylen":8,"magic":"ClientRequest","opaque":0,"opcode":"CREATE_BUCKET","vbucket":0},"peer":{"ip":"127.0.0.1","port":51002},"response":"Success","trace":"request=683347059500:7625520 execute=683347059500:121 execute=690972561500:18","worker_tid":6248}
|
During this time, analytics attempted to get info on the bucket via memcached, which failed with bucket not found:
2022-04-07T08:46:39.188-07:00 DEBU CBAS.netty.DcpConnectHandler [nioEventLoopGroup-2-1] Failed in the SELECT step due to bucket bucket-0 not found (0x1: Key not found)
|
As a result, analytics correctly truncates the datasets associated with the bucket. Should the connect link be tried again, after memcached realizes it exists, I would expect the connect link to succeed, and the datasets to be reingested.
If this is not a satisfactory answer, please reassign to couchbase-bucket for further comment.
| Assignee | Till Westmann [ till ] | Umang [ JIRAUSER24787 ] |
| Resolution | Not a Bug [ 10200 ] | |
| Status | Open [ 1 ] | Resolved [ 5 ] |
Retested the failing scenario with some sleep for nodes to recover after killing memcachced and the test worked, hence closing this issue.
| Status | Resolved [ 5 ] | Closed [ 6 ] |
Last passed on RC3.