Details
Critical Description
A expect a meta data bucket dedicated to eventing (20 functions) should have 1024*20 items or 10,240 ,where each item is about 1318 bytes so the actual data I expect to be in the meta data bucket is 12.87Mb.
However once I deploy all 20 functions and the complete and the system is idle / quiescent system the disk size of the meta data bucket fluctuates in a cycle between 49MB and 170MB
Initial setup for the test case is via instructions found at https://eventing-workshop.s3.amazonaws.com/Instructions.pdf
Now we will create create and deploy 20 functions of 8 workers each as follows based on
simple_08w_inst01
|
simple_08w_inst02
|
simple_08w_inst03
|
+
|
+
|
simple_08w_inst20
|
Each function will have the following identical JS code
function OnUpdate(doc, meta) {
|
/* Check if further analysis is needed */ |
if (doc.type !== "transaction") return; |
var card = register["card:" + doc.card]; |
if (card.threshold > doc.amount) return; |
|
log(meta.id);
|
}
|
Once all 5405 items have been processed by all the "duplicate" functions and there is no backlog on a quiescent system. We seem to have lots of disk activity and the amount of data on disk seems to wildly fluctuate in the meta data bucket "meta".
Now we look at the disk stats across time ( first with two versions of curl )
while [ 1 != 0 ] |
do |
curl --user "$CB_USERNAME:$CB_PASSWORD" --silent "http://admin:jtester@localhost:8091/pools/default/buckets/meta?basic_stats=true" | jq '' | egrep -i diskUsed\|dataUsed\|itemCo\|'"name":' |
|
|
curl --user "$CB_USERNAME:$CB_PASSWORD" --silent "http://localhost:8091/pools/default/buckets/meta/stats" | \ |
jq '.op.samples | \{ curr_items_tot: .curr_items_tot[-1], couch_docs_actual_disk_size: .couch_docs_actual_disk_size[-1], couch_docs_disk_size: .couch_docs_disk_size[-1] }' |
|
|
echo
|
sleep 10 |
done
|
The output follows note the radical size changes to the 'meta' bucket that is dedicated to eventing. It only has the expected 1024*20 items or 10,240 where ech item is about 1318 bytes so the actual data I expect to be 12.87Mb
But as you can see the 'meta' size is all over the place 47MB to 170MB
"name": "meta",
|
"itemCount": 20480,
|
"diskUsed": 82582761,
|
"dataUsed": 83074048,
|
{
|
"curr_items_tot": 20480,
|
"couch_docs_actual_disk_size": 85650665,
|
"couch_docs_disk_size": 89279488
|
}
|
|
|
"name": "meta",
|
"itemCount": 20480,
|
"diskUsed": 89816297,
|
"dataUsed": 96377856,
|
{
|
"curr_items_tot": 20480,
|
"couch_docs_actual_disk_size": 97459433,
|
"couch_docs_disk_size": 103181312
|
}
|
|
|
"name": "meta",
|
"itemCount": 20480,
|
"diskUsed": 106163439,
|
"dataUsed": 106130432,
|
{
|
"curr_items_tot": 20480,
|
"couch_docs_actual_disk_size": 106163439,
|
"couch_docs_disk_size": 106130432
|
}
|
|
|
"name": "meta",
|
"itemCount": 20480,
|
"diskUsed": 110259439,
|
"dataUsed": 110226432,
|
{
|
"curr_items_tot": 20480,
|
"couch_docs_actual_disk_size": 114343151,
|
"couch_docs_disk_size": 125991936
|
}
|
|
|
"name": "meta",
|
"itemCount": 20480,
|
"diskUsed": 129346799,
|
"dataUsed": 130870272,
|
{
|
"curr_items_tot": 20480,
|
"couch_docs_actual_disk_size": 134266095,
|
"couch_docs_disk_size": 137337856
|
}
|
|
|
"name": "meta",
|
"itemCount": 20480,
|
"diskUsed": 145050863,
|
"dataUsed": 18993657,
|
{
|
"curr_items_tot": 20480,
|
"couch_docs_actual_disk_size": 145050863,
|
"couch_docs_disk_size": 158256128
|
}
|
|
|
"name": "meta",
|
"itemCount": 20480,
|
"diskUsed": 166477039,
|
"dataUsed": 18983782,
|
{
|
"curr_items_tot": 20480,
|
"couch_docs_actual_disk_size": 166477039,
|
"couch_docs_disk_size": 170494976
|
}
|
|
|
"name": "meta",
|
"itemCount": 20480,
|
"diskUsed": 163585263,
|
"dataUsed": 101493760,
|
{
|
"curr_items_tot": 20480,
|
"couch_docs_actual_disk_size": 163585263,
|
"couch_docs_disk_size": 101493760
|
}
|
|
|
"name": "meta",
|
"itemCount": 20480,
|
"diskUsed": 44174572,
|
"dataUsed": 47070208,
|
{
|
"curr_items_tot": 20480,
|
"couch_docs_actual_disk_size": 44174572,
|
"couch_docs_disk_size": 47070208
|
}
|
|
|
"name": "meta",
|
"itemCount": 20480,
|
"diskUsed": 52538604,
|
"dataUsed": 67832832,
|
{
|
"curr_items_tot": 20480,
|
"couch_docs_actual_disk_size": 52538604,
|
"couch_docs_disk_size": 67832832
|
}
|
The actual disk via 'du -sk' shows similar behavior as above - remember this system is queasiest it is not processing any new data and there is no eventing backlog. Here we see 49MB to 170MB for the size of meta it is just sort of cycling back and forth.
linuxbrew@couch01:~/scale_it$ cat du.sh
|
|
|
while [ 1!=0 ];
|
do
|
du -sk /space/linuxbrew/couch/install/var/lib/couchbase/data/meta | paste -d " " - - - - - - ;
|
sleep 10;
|
done
|
|
|
linuxbrew@couch01:~/scale_it$ sh du.sh
|
90680 /space/linuxbrew/couch/install/var/lib/couchbase/data/meta
|
97832 /space/linuxbrew/couch/install/var/lib/couchbase/data/meta
|
113976 /space/linuxbrew/couch/install/var/lib/couchbase/data/meta
|
117952 /space/linuxbrew/couch/install/var/lib/couchbase/data/meta
|
136804 /space/linuxbrew/couch/install/var/lib/couchbase/data/meta
|
145876 /space/linuxbrew/couch/install/var/lib/couchbase/data/meta
|
169968 /space/linuxbrew/couch/install/var/lib/couchbase/data/meta
|
177012 /space/linuxbrew/couch/install/var/lib/couchbase/data/meta
|
78308 /space/linuxbrew/couch/install/var/lib/couchbase/data/meta
|
49832 /space/linuxbrew/couch/install/var/lib/couchbase/data/meta
|
70500 /space/linuxbrew/couch/install/var/lib/couchbase/data/meta
|
80140 /space/linuxbrew/couch/install/var/lib/couchbase/data/meta
|
104520 /space/linuxbrew/couch/install/var/lib/couchbase/data/meta
|
^C
|
linuxbrew@couch01:~/scale_it$
|
In addition I am including a few top outputs on this 64MB system showing the memory (real and virtual used), a concern here is that Eventing is using too much real memory
linuxbrew@couch01:~/scale_it$ top -b -n 1 -o RES | head -20
|
top - 20:02:04 up 24 days, 7:00, 6 users, load average: 7.47, 7.29, 8.44
|
Tasks: 474 total, 1 running, 473 sleeping, 0 stopped, 0 zombie
|
%Cpu(s): 19.4 us, 3.7 sy, 0.0 ni, 75.2 id, 0.5 wa, 0.0 hi, 1.2 si, 0.0 st
|
MiB Mem : 64319.3 total, 25178.7 free, 10078.2 used, 29062.4 buff/cache
|
MiB Swap: 65433.0 total, 65429.5 free, 3.5 used. 53870.2 avail Mem
|
|
|
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
|
31921 linuxbr+ 20 0 15.4g 1.7g 19580 S 58.8 2.6 125:27.87 eventing-produc
|
31620 linuxbr+ 20 0 7772432 1.4g 7692 S 223.5 2.2 231:17.32 beam.smp
|
31938 linuxbr+ 20 0 8639016 320304 13056 S 158.8 0.5 130:32.66 memcached
|
31711 linuxbr+ 20 0 5978088 125616 17124 S 0.0 0.2 0:47.81 beam.smp
|
31829 linuxbr+ 20 0 2462092 118160 21644 S 29.4 0.2 24:51.83 indexer
|
31856 linuxbr+ 20 0 1519344 99872 15012 S 5.9 0.2 1:09.20 projector
|
31889 linuxbr+ 20 0 4216224 78904 34804 S 5.9 0.1 0:49.31 cbq-engine
|
31433 linuxbr+ 20 0 6164320 54032 7144 S 0.0 0.1 2:04.62 beam.smp
|
2587 linuxbr+ 20 0 1383880 53852 23476 S 0.0 0.1 0:21.08 eventing-consum
|
1077 linuxbr+ 20 0 1383880 53788 23408 S 0.0 0.1 0:21.01 eventing-consum
|
4782 linuxbr+ 20 0 1383880 53784 23408 S 0.0 0.1 0:17.94 eventing-consum
|
6122 linuxbr+ 20 0 1383880 53764 23372 S 0.0 0.1 0:16.77 eventing-consum
|
4781 linuxbr+ 20 0 1383880 53760 23372 S 0.0 0.1 0:17.93 eventing-consum
|
linuxbrew@couch01:~/scale_it$ top -b -n 1 -o VIRT | head -20
|
top - 20:02:16 up 24 days, 7:00, 6 users, load average: 6.61, 7.11, 8.36
|
Tasks: 474 total, 1 running, 473 sleeping, 0 stopped, 0 zombie
|
%Cpu(s): 16.6 us, 3.0 sy, 0.0 ni, 79.4 id, 0.0 wa, 0.0 hi, 0.9 si, 0.0 st
|
MiB Mem : 64319.3 total, 25241.8 free, 10068.8 used, 29008.6 buff/cache
|
MiB Swap: 65433.0 total, 65429.5 free, 3.5 used. 53879.3 avail Mem
|
|
|
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
|
31921 linuxbr+ 20 0 15.4g 1.7g 19580 S 225.0 2.6 125:42.67 eventing-produc
|
31938 linuxbr+ 20 0 8639016 318836 13056 S 156.2 0.5 130:53.36 memcached
|
31620 linuxbr+ 20 0 7754584 1.3g 7692 S 37.5 2.1 231:36.05 beam.smp
|
31433 linuxbr+ 20 0 6164320 54032 7144 S 0.0 0.1 2:04.71 beam.smp
|
31711 linuxbr+ 20 0 5978088 125148 17124 S 0.0 0.2 0:47.91 beam.smp
|
31889 linuxbr+ 20 0 4216224 78904 34804 S 6.2 0.1 0:49.50 cbq-engine
|
31829 linuxbr+ 20 0 2462092 118160 21644 S 37.5 0.2 24:55.30 indexer
|
31856 linuxbr+ 20 0 1519344 99872 15012 S 0.0 0.2 1:09.36 projector
|
534 linuxbr+ 20 0 1383880 52980 23136 S 0.0 0.1 0:21.80 eventing-consum
|
539 linuxbr+ 20 0 1383880 53436 23316 S 0.0 0.1 0:22.40 eventing-consum
|
540 linuxbr+ 20 0 1383880 53404 23024 S 6.2 0.1 0:22.82 eventing-consum
|
549 linuxbr+ 20 0 1383880 53300 23452 S 0.0 0.1 0:22.41 eventing-consum
|
570 linuxbr+ 20 0 1383880 53476 23360 S 6.2 0.1 0:22.84 eventing-consum
|
linuxbrew@couch01:~/scale_it$ top -b -n 1 -o VIRT | grep eventing-consum | wc -l
|
160
|
linuxbrew@couch01:~/scale_it$
|