Details
-
Bug
-
Resolution: Unresolved
-
Major
-
Columnar 1.0.0
-
1.0.0-2230
-
Untriaged
-
0
-
Unknown
Description
Workload -
Type | Number of collections | Number of items in millions | Total count in millions |
---|---|---|---|
Remote | 80 | 75 | 6000 |
Standalone | 50 | 8 | 4000* |
Kafka | 30 | 33.5 | ~1000 |
The change from the previous runs has been the increase in the number of Kafka collections.
*Some standalone collections have 8 mil and some have multiples of 8 million items. The total doc count is 4000 million ( 4 billion) items.
Number of links = 6 ( 2 remote + 2 external + 2 kafka). 1 remote link and 1 kafka link is active.
OOM seen on 003 and 004 -
003 -
2024-07-21T05:24:12.044+00:00 INFO CBAS.adapter.TopicRecordReader [SAO:JID:0.4085:TAID:TID:ANID:ODID:92:0:9:0:(Default.linkWCvvGTPb.b-3-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196,b-2-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196,b-1-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196(CB))[9]:TO] State for Topic testSystemNewesttXXUnITt.systemTest.systemTestCollectionLarge is [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] |
2024-07-21T05:24:12.044+00:00 INFO CBAS.adapter.TopicRecordReader [SAO:JID:0.4085:TAID:TID:ANID:ODID:92:0:5:0:(Default.linkWCvvGTPb.b-3-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196,b-2-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196,b-1-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196(CB))[5]:TO] State for Topic testSystemNewesttXXUnITt.systemTest.systemTestCollectionLarge is [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] |
2024-07-21T05:24:12.044+00:00 INFO CBAS.adapter.TopicRecordReader [SAO:JID:0.4085:TAID:TID:ANID:ODID:92:0:7:0:(Default.linkWCvvGTPb.b-3-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196,b-2-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196,b-1-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196(CB))[7]:TO] State for Topic testSystemNewesttXXUnITt.systemTest.systemTestCollectionLarge is [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] |
2024-07-21T05:24:12.044+00:00 INFO CBAS.adapter.TopicRecordReader [SAO:JID:0.4085:TAID:TID:ANID:ODID:92:0:19:0:(Default.linkWCvvGTPb.b-3-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196,b-2-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196,b-1-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196(CB))[19]:TO] State for Topic testSystemNewesttXXUnITt.systemTest.systemTestCollectionLarge is [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] |
2024-07-21T05:24:12.044+00:00 INFO CBAS.adapter.TopicRecordReader [SAO:JID:0.4085:TAID:TID:ANID:ODID:92:0:25:0:(Default.linkWCvvGTPb.b-3-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196,b-2-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196,b-1-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196(CB))[25]:TO] State for Topic testSystemNewesttXXUnITt.systemTest.systemTestCollectionLarge is [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] |
2024-07-21T05:24:12.045+00:00 INFO CBAS.adapter.TopicRecordReader [SAO:JID:0.4085:TAID:TID:ANID:ODID:92:0:12:0:(Default.linkWCvvGTPb.b-3-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196,b-2-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196,b-1-public.qekafkatestcluster.7b9vtv.c13.kafka.us-east-1.amazonaws.com:9196(CB))[12]:TO] State for Topic testSystemNewesttXXUnITt.systemTest.systemTestCollectionLarge is [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] |
2024-07-21T05:24:12.049+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-7925:d5162036730bed9965768aa18a9cdbe4] Data Polling started... |
2024-07-21T05:24:12.050+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-7940:d5162036730bed9965768aa18a9cdbe4] Data Polling started... |
2024-07-21T05:24:12.050+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-7946:d5162036730bed9965768aa18a9cdbe4] Data Polling started... |
2024-07-21T05:24:12.050+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-7949:d5162036730bed9965768aa18a9cdbe4] Data Polling started... |
2024-07-21T05:24:12.050+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-7965:d5162036730bed9965768aa18a9cdbe4] Data Polling started... |
2024-07-21T05:24:12.050+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-7963:d5162036730bed9965768aa18a9cdbe4] Data Polling started... |
2024-07-21T05:24:12.050+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-7928:d5162036730bed9965768aa18a9cdbe4] Data Polling started... |
2024-07-21T05:24:12.050+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-7964:d5162036730bed9965768aa18a9cdbe4] Data Polling started... |
2024-07-21T05:24:12.050+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-7966:d5162036730bed9965768aa18a9cdbe4] Data Polling started... |
2024-07-21T05:24:12.050+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-7967:d5162036730bed9965768aa18a9cdbe4] Data Polling started... |
Terminating due to java.lang.OutOfMemoryError: Java heap space
|
2024-07-21T05:25:18.382+00:00 WARN CBAS.cbas analytics driver has exited w/ exit status 3 |
2024-07-21T05:25:18.393+00:00 INFO CBAS.cbas exiting |
004
2024-07-21T05:25:24.262+00:00 INFO CBAS.runtime.TopicOffsetUpdateCallback [Executor-8218:c44c76108444df41c0cc5d85111d4b7b] triggering flush if needed on {"dir" : "/var/cb-cache/@analytics/v_iodevice_4/storage/partition_100/Database11boluRBGY/scope0btgjupyd/LinkedDatasetskXWsAeXnf/0/LinkedDatasetskXWsAeXnf", "memory" : [{"state":"INACTIVE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[1,1]", "index":{"class": "BTree", "file": "storage/partition_100/Database11boluRBGY/scope0btgjupyd/LinkedDatasetskXWsAeXnf/0/LinkedDatasetskXWsAeXnf_virtual_0"}}, {"state":"READABLE_WRITABLE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[2,2]", "index":{"class": "BTree", "file": "storage/partition_100/Database11boluRBGY/scope0btgjupyd/LinkedDatasetskXWsAeXnf/0/LinkedDatasetskXWsAeXnf_virtual_1"}}], "disk" : 1, "num-scheduled-flushes":0, "current-memory-component":1} to persist the topic state. reason stopping ingestion |
2024-07-21T05:25:24.262+00:00 INFO CBAS.runtime.TopicOffsetUpdateCallback [Executor-8228:c44c76108444df41c0cc5d85111d4b7b] triggering flush if needed on {"dir" : "/var/cb-cache/@analytics/v_iodevice_12/storage/partition_108/Database19EtVbaMXn/scope1pAMKvKXz/LinkedDatasetuUroaejcPK/0/LinkedDatasetuUroaejcPK", "memory" : [{"state":"INACTIVE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[1,1]", "index":{"class": "BTree", "file": "storage/partition_108/Database19EtVbaMXn/scope1pAMKvKXz/LinkedDatasetuUroaejcPK/0/LinkedDatasetuUroaejcPK_virtual_0"}}, {"state":"READABLE_WRITABLE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[2,2]", "index":{"class": "BTree", "file": "storage/partition_108/Database19EtVbaMXn/scope1pAMKvKXz/LinkedDatasetuUroaejcPK/0/LinkedDatasetuUroaejcPK_virtual_1"}}], "disk" : 1, "num-scheduled-flushes":0, "current-memory-component":1} to persist the topic state. reason stopping ingestion |
2024-07-21T05:25:24.262+00:00 INFO CBAS.runtime.TopicOffsetUpdateCallback [Executor-8305:c44c76108444df41c0cc5d85111d4b7b] triggering flush if needed on {"dir" : "/var/cb-cache/@analytics/v_iodevice_10/storage/partition_122/Database16DbijEPqn/scope0SmnYSzlx/LinkedDatasetGzIKCMTSWg/0/LinkedDatasetGzIKCMTSWg", "memory" : [{"state":"READABLE_WRITABLE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[1,1]", "index":{"class": "BTree", "file": "storage/partition_122/Database16DbijEPqn/scope0SmnYSzlx/LinkedDatasetGzIKCMTSWg/0/LinkedDatasetGzIKCMTSWg_virtual_0"}}, {"state":"INACTIVE", "writers":0, "readers":0, "pendingFlushes":0, "id":"null", "index":{"class": "BTree", "file": "storage/partition_122/Database16DbijEPqn/scope0SmnYSzlx/LinkedDatasetGzIKCMTSWg/0/LinkedDatasetGzIKCMTSWg_virtual_1"}}], "disk" : 0, "num-scheduled-flushes":0, "current-memory-component":0} to persist the topic state. reason stopping ingestion |
2024-07-21T05:25:24.262+00:00 INFO CBAS.runtime.TopicOffsetUpdateCallback [Executor-8226:c44c76108444df41c0cc5d85111d4b7b] triggering flush if needed on {"dir" : "/var/cb-cache/@analytics/v_iodevice_6/storage/partition_86/Database22UkFgeWkt/scope1XJjfUFVy/LinkedDatasetyzcsmPheLH/0/LinkedDatasetyzcsmPheLH", "memory" : [{"state":"INACTIVE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[1,1]", "index":{"class": "BTree", "file": "storage/partition_86/Database22UkFgeWkt/scope1XJjfUFVy/LinkedDatasetyzcsmPheLH/0/LinkedDatasetyzcsmPheLH_virtual_0"}}, {"state":"READABLE_WRITABLE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[2,2]", "index":{"class": "BTree", "file": "storage/partition_86/Database22UkFgeWkt/scope1XJjfUFVy/LinkedDatasetyzcsmPheLH/0/LinkedDatasetyzcsmPheLH_virtual_1"}}], "disk" : 1, "num-scheduled-flushes":0, "current-memory-component":1} to persist the topic state. reason stopping ingestion |
2024-07-21T05:25:24.262+00:00 INFO CBAS.runtime.TopicOffsetUpdateCallback [Executor-8319:c44c76108444df41c0cc5d85111d4b7b] triggering flush if needed on {"dir" : "/var/cb-cache/@analytics/v_iodevice_10/storage/partition_106/Database16DbijEPqn/scope0SmnYSzlx/LinkedDatasetGzIKCMTSWg/0/LinkedDatasetGzIKCMTSWg", "memory" : [{"state":"INACTIVE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[1,1]", "index":{"class": "BTree", "file": "storage/partition_106/Database16DbijEPqn/scope0SmnYSzlx/LinkedDatasetGzIKCMTSWg/0/LinkedDatasetGzIKCMTSWg_virtual_0"}}, {"state":"READABLE_WRITABLE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[2,2]", "index":{"class": "BTree", "file": "storage/partition_106/Database16DbijEPqn/scope0SmnYSzlx/LinkedDatasetGzIKCMTSWg/0/LinkedDatasetGzIKCMTSWg_virtual_1"}}], "disk" : 1, "num-scheduled-flushes":0, "current-memory-component":1} to persist the topic state. reason stopping ingestion |
2024-07-21T05:25:24.817+00:00 INFO CBAS.runtime.TopicOffsetUpdateCallback [Executor-8149:c44c76108444df41c0cc5d85111d4b7b] triggering flush if needed on {"dir" : "/var/cb-cache/@analytics/v_iodevice_10/storage/partition_106/Database19EtVbaMXn/scope1pAMKvKXz/LinkedDatasetuUroaejcPK/0/LinkedDatasetuUroaejcPK", "memory" : [{"state":"INACTIVE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[1,1]", "index":{"class": "BTree", "file": "storage/partition_106/Database19EtVbaMXn/scope1pAMKvKXz/LinkedDatasetuUroaejcPK/0/LinkedDatasetuUroaejcPK_virtual_0"}}, {"state":"READABLE_WRITABLE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[2,2]", "index":{"class": "BTree", "file": "storage/partition_106/Database19EtVbaMXn/scope1pAMKvKXz/LinkedDatasetuUroaejcPK/0/LinkedDatasetuUroaejcPK_virtual_1"}}], "disk" : 1, "num-scheduled-flushes":0, "current-memory-component":1} to persist the topic state. reason stopping ingestion |
2024-07-21T05:25:26.972+00:00 INFO CBAS.runtime.TopicOffsetUpdateCallback [Executor-8235:c44c76108444df41c0cc5d85111d4b7b] triggering flush if needed on {"dir" : "/var/cb-cache/@analytics/v_iodevice_10/storage/partition_90/Database9TdLzCFWt/scope1CdoPyywe/LinkedDatasetkiwYjiUuzm/0/LinkedDatasetkiwYjiUuzm", "memory" : [{"state":"INACTIVE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[1,1]", "index":{"class": "BTree", "file": "storage/partition_90/Database9TdLzCFWt/scope1CdoPyywe/LinkedDatasetkiwYjiUuzm/0/LinkedDatasetkiwYjiUuzm_virtual_0"}}, {"state":"READABLE_WRITABLE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[2,2]", "index":{"class": "BTree", "file": "storage/partition_90/Database9TdLzCFWt/scope1CdoPyywe/LinkedDatasetkiwYjiUuzm/0/LinkedDatasetkiwYjiUuzm_virtual_1"}}], "disk" : 1, "num-scheduled-flushes":0, "current-memory-component":1} to persist the topic state. reason stopping ingestion |
2024-07-21T05:25:26.972+00:00 INFO CBAS.runtime.TopicOffsetUpdateCallback [Executor-8239:c44c76108444df41c0cc5d85111d4b7b] triggering flush if needed on {"dir" : "/var/cb-cache/@analytics/v_iodevice_6/storage/partition_86/Database19EtVbaMXn/scope0yKneyzBN/LinkedDatasetjBTKHjyPLf/0/LinkedDatasetjBTKHjyPLf", "memory" : [{"state":"INACTIVE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[1,1]", "index":{"class": "BTree", "file": "storage/partition_86/Database19EtVbaMXn/scope0yKneyzBN/LinkedDatasetjBTKHjyPLf/0/LinkedDatasetjBTKHjyPLf_virtual_0"}}, {"state":"READABLE_WRITABLE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[2,2]", "index":{"class": "BTree", "file": "storage/partition_86/Database19EtVbaMXn/scope0yKneyzBN/LinkedDatasetjBTKHjyPLf/0/LinkedDatasetjBTKHjyPLf_virtual_1"}}], "disk" : 1, "num-scheduled-flushes":0, "current-memory-component":1} to persist the topic state. reason stopping ingestion |
Terminating due to java.lang.OutOfMemoryError: Java heap space
|
At the time of this error, only Kafka ingestion was going on as far as I can analyse. There was no query/mutation workload. Ingestion via remote links was already complete.
Marking this as Major as the cluster was still usable after this. But if RCA deems this critical, please mark it up.
cbcollect ->
Supportal -> https://supportal.couchbase.com/snapshot/6221cab353b5dfd0fa019da9948047b8::20