Details
-
Bug
-
Resolution: User Error
-
Major
-
5.5.0
-
Untriaged
-
-
Unknown
Description
Build : 5.5.0-2671
Test : 2i Component System Test
The query service on one of the query nodes was killed by OOM killer. Logs attached. Seeing the following error on diag logs:
Service 'query' exited with status 137. Restarting. Messages:
|
2018-05-08T02:06:30.796-07:00 [Info] [Queryport-connpool:172.23.104.23:9101] open new connection ...
|
2018-05-08T02:06:30.887-07:00 [Info] GSIC[default/other-3-1525761116660419159] request(3bc860c0-1c2c-4244-bf91-200de41e0e18) removing backfill file /opt/couchbase/var/lib/couchbase/tmp/scan-results21677152890392 ...
|
2018-05-08T02:06:30.914-07:00 [Info] GSIC[default/other-1-1525761037634374335] request(53491a03-4700-4a31-9650-8acaab0580cb) removing backfill file /opt/couchbase/var/lib/couchbase/tmp/scan-results21677389195357 ...
|
2018-05-08T02:06:31.004-07:00 [Info] GSIC[default/other-1-1525761037634374335] a75ae2fc-4ffb-476b-8044-01e60edb9fdc new backfill file ... /opt/couchbase/var/lib/couchbase/tmp/scan-results21677735781404
|
2018-05-08T02:06:31.032-07:00 [Info] GSIC[default/other-3-1525761116660419159] cb15c87e-6bcc-45c1-a3e5-7d50f6f3ac75 new backfill file ... /opt/couchbase/var/lib/couchbase/tmp/scan-results21677120074955
|
[goport(/opt/couchbase/bin/cbq-engine)] 2018/05/08 02:06:44 child process exited with status 137
|
From dmesg -T:
[Tue May 8 02:06:40 2018] [21677] 1000 21677 5995070 5689875 11219 0 0 cbq-engine
[Tue May 8 02:06:40 2018] [25087] 89 25087 22412 250 45 0 0 pickup
[Tue May 8 02:06:40 2018] Out of memory: Kill process 21677 (cbq-engine) score 930 or sacrifice child
[Tue May 8 02:06:40 2018] Killed process 21677 (cbq-engine) total-vm:23980280kB, anon-rss:22759500kB, file-rss:0kB, shmem-rss:0kB