Details
-
Bug
-
Resolution: Won't Fix
-
Major
-
5.5.0
-
CentOS 7.4
E5-2680 v3 (48 vCPU)
-
Untriaged
-
Centos 64-bit
-
Please navigate from http://showfast.sc.couchbase.com/#/timeline/Linux/n1ql/Q1_Q3_thr/KV
-
No
Description
Setup:
- 4 KV nodes, 1 Query node
- 1 bucket
- Read-only workload
- 960 client threads (running on two separate client boxes)
I am going to compare two similar queries:
SELECT * FROM `bucket-1` USE KEYS[$1];"
|
and
SELECT META().xattrs.`user` FROM `bucket-1` USE KEYS[$1];"
|
Actual documents and XATTR objects ("user") have absolutely the same content.
This is what we observe:
- Regular fetches: up to 84K queries/sec
- XATTR fetches: up to 67K queries/sec (20% difference)
Usually SUBDOC requests are slightly slower than normal GET requests. Here is comparison of KV timings:
The following data is collected for "GET"
|
[ 0 - 9]us ( 74.19%) 18593206 | ############################################
|
[ 10 - 19]us ( 99.01%) 6218721 | ##############
|
[ 20 - 29]us ( 99.88%) 219340 |
|
[ 30 - 39]us ( 99.97%) 22385 |
|
[ 40 - 49]us ( 99.99%) 3913 |
|
[ 50 - 59]us ( 99.99%) 698 |
|
[ 60 - 69]us ( 99.99%) 321 |
|
[ 70 - 79]us ( 99.99%) 127 |
|
[ 80 - 89]us ( 99.99%) 55 |
|
[ 90 - 99]us ( 99.99%) 29 |
|
[ 100 - 109]us ( 99.99%) 18 |
|
[ 110 - 119]us ( 99.99%) 41 |
|
[ 120 - 129]us ( 99.99%) 35 |
|
[ 130 - 139]us ( 99.99%) 49 |
|
[ 140 - 149]us ( 99.99%) 60 |
|
[ 150 - 159]us ( 99.99%) 271 |
|
[ 160 - 169]us (100.00%) 286 |
|
[ 170 - 179]us (100.00%) 155 |
|
[ 180 - 189]us (100.00%) 74 |
|
[ 190 - 199]us (100.00%) 49 |
|
[ 200 - 209]us (100.00%) 47 |
|
[ 210 - 219]us (100.00%) 31 |
|
[ 220 - 229]us (100.00%) 32 |
|
[ 230 - 239]us (100.00%) 31 |
|
[ 240 - 249]us (100.00%) 36 |
|
[ 250 - 259]us (100.00%) 26 |
|
[ 260 - 269]us (100.00%) 23 |
|
[ 270 - 279]us (100.00%) 26 |
|
[ 280 - 289]us (100.00%) 40 |
|
[ 290 - 299]us (100.00%) 24 |
|
[ 300 - 309]us (100.00%) 27 |
|
[ 310 - 319]us (100.00%) 34 |
|
[ 320 - 329]us (100.00%) 35 |
|
[ 330 - 339]us (100.00%) 24 |
|
[ 340 - 349]us (100.00%) 20 |
|
[ 350 - 359]us (100.00%) 15 |
|
[ 360 - 369]us (100.00%) 18 |
|
[ 370 - 379]us (100.00%) 25 |
|
[ 380 - 389]us (100.00%) 16 |
|
[ 390 - 399]us (100.00%) 7 |
|
[ 400 - 409]us (100.00%) 17 |
|
[ 410 - 419]us (100.00%) 13 |
|
[ 420 - 429]us (100.00%) 15 |
|
[ 430 - 439]us (100.00%) 16 |
|
[ 440 - 449]us (100.00%) 13 |
|
[ 450 - 459]us (100.00%) 11 |
|
[ 460 - 469]us (100.00%) 6 |
|
[ 470 - 479]us (100.00%) 10 |
|
[ 480 - 489]us (100.00%) 7 |
|
[ 490 - 499]us (100.00%) 8 |
|
[ 500 - 509]us (100.00%) 8 |
|
[ 510 - 519]us (100.00%) 8 |
|
[ 520 - 529]us (100.00%) 12 |
|
[ 530 - 539]us (100.00%) 10 |
|
[ 540 - 549]us (100.00%) 8 |
|
[ 550 - 559]us (100.00%) 7 |
|
[ 560 - 569]us (100.00%) 9 |
|
[ 570 - 579]us (100.00%) 7 |
|
[ 580 - 589]us (100.00%) 4 |
|
[ 590 - 599]us (100.00%) 1 |
|
[ 600 - 609]us (100.00%) 7 |
|
[ 610 - 619]us (100.00%) 1 |
|
[ 620 - 629]us (100.00%) 4 |
|
[ 630 - 639]us (100.00%) 2 |
|
[ 640 - 649]us (100.00%) 4 |
|
[ 650 - 659]us (100.00%) 1 |
|
[ 660 - 669]us (100.00%) 1 |
|
[ 670 - 679]us (100.00%) 1 |
|
[ 680 - 689]us (100.00%) 1 |
|
[ 700 - 709]us (100.00%) 1 |
|
[ 710 - 719]us (100.00%) 1 |
|
[ 740 - 749]us (100.00%) 3 |
|
[ 750 - 759]us (100.00%) 1 |
|
[ 760 - 769]us (100.00%) 1 |
|
[ 770 - 779]us (100.00%) 1 |
|
[ 790 - 799]us (100.00%) 1 |
|
[ 800 - 809]us (100.00%) 3 |
|
[ 810 - 819]us (100.00%) 4 |
|
[ 820 - 829]us (100.00%) 1 |
|
[ 830 - 839]us (100.00%) 2 |
|
[ 880 - 889]us (100.00%) 2 |
|
[ 900 - 909]us (100.00%) 1 |
|
[ 930 - 939]us (100.00%) 1 |
|
[ 940 - 949]us (100.00%) 1 |
|
[ 950 - 959]us (100.00%) 1 |
|
[ 960 - 969]us (100.00%) 1 |
|
[ 970 - 979]us (100.00%) 2 |
|
[ 980 - 989]us (100.00%) 1 |
|
[ 1 - 1]ms (100.00%) 10 |
|
Total: 25060621 operations
|
The following data is collected for "SUBDOC_MULTI_LOOKUP"
|
[ 20 - 29]us ( 0.00%) 9 |
|
[ 30 - 39]us ( 48.56%) 9820216 | ############################################
|
[ 40 - 49]us ( 78.44%) 6042809 | ###########################
|
[ 50 - 59]us ( 95.82%) 3515808 | ###############
|
[ 60 - 69]us ( 98.91%) 625144 | ##
|
[ 70 - 79]us ( 99.59%) 137577 |
|
[ 80 - 89]us ( 99.74%) 31079 |
|
[ 90 - 99]us ( 99.84%) 19812 |
|
[ 100 - 109]us ( 99.91%) 13942 |
|
[ 110 - 119]us ( 99.94%) 6372 |
|
[ 120 - 129]us ( 99.95%) 2454 |
|
[ 130 - 139]us ( 99.96%) 1274 |
|
[ 140 - 149]us ( 99.97%) 810 |
|
[ 150 - 159]us ( 99.97%) 605 |
|
[ 160 - 169]us ( 99.97%) 465 |
|
[ 170 - 179]us ( 99.97%) 284 |
|
[ 180 - 189]us ( 99.97%) 319 |
|
[ 190 - 199]us ( 99.98%) 419 |
|
[ 200 - 209]us ( 99.98%) 552 |
|
[ 210 - 219]us ( 99.98%) 617 |
|
[ 220 - 229]us ( 99.98%) 427 |
|
[ 230 - 239]us ( 99.98%) 305 |
|
[ 240 - 249]us ( 99.99%) 182 |
|
[ 250 - 259]us ( 99.99%) 157 |
|
[ 260 - 269]us ( 99.99%) 123 |
|
[ 270 - 279]us ( 99.99%) 125 |
|
[ 280 - 289]us ( 99.99%) 98 |
|
[ 290 - 299]us ( 99.99%) 110 |
|
[ 300 - 309]us ( 99.99%) 111 |
|
[ 310 - 319]us ( 99.99%) 97 |
|
[ 320 - 329]us ( 99.99%) 95 |
|
[ 330 - 339]us ( 99.99%) 100 |
|
[ 340 - 349]us ( 99.99%) 81 |
|
[ 350 - 359]us ( 99.99%) 116 |
|
[ 360 - 369]us ( 99.99%) 112 |
|
[ 370 - 379]us ( 99.99%) 127 |
|
[ 380 - 389]us ( 99.99%) 106 |
|
[ 390 - 399]us ( 99.99%) 69 |
|
[ 400 - 409]us ( 99.99%) 59 |
|
[ 410 - 419]us ( 99.99%) 47 |
|
[ 420 - 429]us ( 99.99%) 44 |
|
[ 430 - 439]us ( 99.99%) 59 |
|
[ 440 - 449]us (100.00%) 53 |
|
[ 450 - 459]us (100.00%) 41 |
|
[ 460 - 469]us (100.00%) 47 |
|
[ 470 - 479]us (100.00%) 43 |
|
[ 480 - 489]us (100.00%) 41 |
|
[ 490 - 499]us (100.00%) 44 |
|
[ 500 - 509]us (100.00%) 34 |
|
[ 510 - 519]us (100.00%) 37 |
|
[ 520 - 529]us (100.00%) 34 |
|
[ 530 - 539]us (100.00%) 25 |
|
[ 540 - 549]us (100.00%) 29 |
|
[ 550 - 559]us (100.00%) 30 |
|
[ 560 - 569]us (100.00%) 28 |
|
[ 570 - 579]us (100.00%) 24 |
|
[ 580 - 589]us (100.00%) 32 |
|
[ 590 - 599]us (100.00%) 24 |
|
[ 600 - 609]us (100.00%) 27 |
|
[ 610 - 619]us (100.00%) 25 |
|
[ 620 - 629]us (100.00%) 22 |
|
[ 630 - 639]us (100.00%) 22 |
|
[ 640 - 649]us (100.00%) 21 |
|
[ 650 - 659]us (100.00%) 21 |
|
[ 660 - 669]us (100.00%) 21 |
|
[ 670 - 679]us (100.00%) 13 |
|
[ 680 - 689]us (100.00%) 14 |
|
[ 690 - 699]us (100.00%) 7 |
|
[ 700 - 709]us (100.00%) 11 |
|
[ 710 - 719]us (100.00%) 14 |
|
[ 720 - 729]us (100.00%) 16 |
|
[ 730 - 739]us (100.00%) 11 |
|
[ 740 - 749]us (100.00%) 9 |
|
[ 750 - 759]us (100.00%) 10 |
|
[ 760 - 769]us (100.00%) 13 |
|
[ 770 - 779]us (100.00%) 5 |
|
[ 780 - 789]us (100.00%) 9 |
|
[ 790 - 799]us (100.00%) 9 |
|
[ 800 - 809]us (100.00%) 7 |
|
[ 810 - 819]us (100.00%) 6 |
|
[ 820 - 829]us (100.00%) 5 |
|
[ 830 - 839]us (100.00%) 8 |
|
[ 840 - 849]us (100.00%) 7 |
|
[ 850 - 859]us (100.00%) 6 |
|
[ 860 - 869]us (100.00%) 4 |
|
[ 870 - 879]us (100.00%) 3 |
|
[ 880 - 889]us (100.00%) 5 |
|
[ 890 - 899]us (100.00%) 1 |
|
[ 900 - 909]us (100.00%) 4 |
|
[ 910 - 919]us (100.00%) 1 |
|
[ 920 - 929]us (100.00%) 2 |
|
[ 930 - 939]us (100.00%) 5 |
|
[ 940 - 949]us (100.00%) 2 |
|
[ 950 - 959]us (100.00%) 3 |
|
[ 960 - 969]us (100.00%) 4 |
|
[ 970 - 979]us (100.00%) 3 |
|
[ 980 - 989]us (100.00%) 2 |
|
[ 990 - 999]us (100.00%) 4 |
|
[ 1 - 1]ms (100.00%) 70 |
|
[ 2 - 2]ms (100.00%) 1 |
|
Total: 20224336 operations
|
However, 99.9% of SUBDOC_MULTI_LOOKUP requests take less than 110us. That is less than network latency anyway. I don't think the difference is crucial.
CPU profiling (see attachments) indicates that Fetch with getSubDocFetchResults path is 2x more expensive than document fetches. There seem to be a higher GC overhead as well.
