goroutine memory leak in query engine execution pipeline

Memory leak discovered via CBSE-3206

Email thread:

From: Gerald Sangudi <Gerald@couchbase.com>
Date: Thursday, 22 December 2016 20:29
To: Marco Greco <marco.greco@couchbase.com>, Sitaram Vemulapalli <Sitaram.Vemulapalli@couchbase.com>
Subject: Re: CBSE-3260
 
Ok, a bit complicated.
 
afterItems() does cleanup, so we cannot skip it.
 
We need to add a boolean field base.stopped.
 
In sendItem(), we need to run in a loop:
 
(1) Remote the first select.
 
(2) Wrap the second select in an infinite loop: while !this.stopped {}
 
(3) Add a default break to the select.
 
Marco,
 
I can make these changes, or you can make them. lmk.
 
Gerald
 
From: Marco Greco <Marco.Greco@couchbase.com>
Date: Thursday, December 22, 2016 at 12:07 PM
To: Gerald Sangudi <gerald@couchbase.com>
Subject: Re: CBSE-3260
 
Hey Gerald – I am being pressed for an update. Did you take a look at Sitaram's suggestion in the end?
Also the CBSE # is 32>>06<< (fat fingers, me?)
 
From: Gerald Sangudi <Gerald@couchbase.com>
Date: Wednesday, 14 December 2016 15:29
To: Marco Greco <marco.greco@couchbase.com>, Sitaram Vemulapalli <Sitaram.Vemulapalli@couchbase.com>, Keshav Murthy <keshav@couchbase.com>
Subject: Re: CBSE-3260
 
I’ll take a look.
 
From: Marco Greco <Marco.Greco@couchbase.com>
Date: Wednesday, December 14, 2016 at 1:36 AM
To: Sitaram Vemulapalli <Sitaram.Vemulapalli@couchbase.com>, Gerald Sangudi <gerald@couchbase.com>, Keshav Murthy <keshav@couchbase.com>
Subject: Re: CBSE-3260
 
I think that's a good suggestion.
Two options – either we could completely ignore afterItems() in rumConsumer in case of a stop signal, or we call afterItems(), but passing it a stop flag and let each individual afterItems() method decide what it should do in case of a stop being received.
I don't have a preference – the second is more granular but more open to leaks. The first is safer but could be heavy handed.
 
From: Sitaram Vemulapalli <Sitaram.Vemulapalli@couchbase.com>
Date: Wednesday, 14 December 2016 07:17
To: Marco Greco <marco.greco@couchbase.com>, Gerald Sangudi <Gerald@couchbase.com>, Keshav Murthy <keshav@couchbase.com>
Subject: Re: CBSE-3260
 
I think the following code should execute only when it is not coming from StopChanel.  Otherwise top level  iterator sends stop and it finishes and  this keep sending items and channel might full.
 
https://github.com/couchbase/query/blob/master/execution/base.go#L197
 
 
Example : Fetch
                  Filter
            Both uses runConsumer, Fetch has afterItems() which does actual fetch  and Filter has dummy afterItems()
            Filter receives stop and it notify’s children and completes it go routine.
            Fetch does actual fetch of remaining batched keys and trying to send to parent which no longer exist, it may blocking on channel.
   
            As we received stop ignore remaining batched keys.
 
 
Regards,
Sitaram
 
 
From: Marco Greco <Marco.Greco@couchbase.com>
Date: Tuesday, December 13, 2016 at 8:44 PM
To: Sitaram Vemulapalli <sitaram.vemulapalli@couchbase.com>, Gerald Sangudi <Gerald@couchbase.com>, Keshav Murthy <keshav@couchbase.com>
Subject: CBSE-3260
 
There is indeed a goroutine leak – I see the remnants of an intersect scan that have not gone away
 
1 @ 0x419925 0x419998 0x407bd2 0x40776b 0xa53864 0xa532cb 0x78a2e5 0x7d7172 0x7d6ad2 0x801ee3 0x5408c4 0x7cb99f 0x7d68f5 0x42d291

1. 0xa53864github.com/couchbase/go-couchbase.(*Bucket).getBulk+0x504/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/go-couchbase/client.go:482
2. 0xa532cbgithub.com/couchbase/go-couchbase.(*Bucket).GetBulk+0x5b/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/go-couchbase/client.go:444
3. 0x78a2e5github.com/couchbase/query/datastore/couchbase.(*keyspace).Fetch+0x105/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:691
4. 0x7d7172github.com/couchbase/query/execution.(*Fetch).flushBatch+0x652/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/execution/fetch.go:116
5. 0x7d6ad2github.com/couchbase/query/execution.(*Fetch).afterItems+0x32/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/execution/fetch.go:66
6. 0x801ee3github.com/couchbase/query/execution.func·004+0x343/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/execution/base.go:192
7. 0x5408c4sync.(*Once).Do+0xd4/usr/local/go/src/sync/once.go:44
8. 0x7cb99fgithub.com/couchbase/query/execution.(*base).runConsumer+0x6f/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/execution/base.go:193
9. 0x7d68f5github.com/couchbase/query/execution.(*Fetch).RunOnce+0x95/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/execution/fetch.go:50
    
   4 @ 0x419925 0x41bbcc 0x41b062 0x7cb917 0x7f4e88 0x808f6a 0x5408c4 0x7f4916 0x42d291
10. 0x7cb917github.com/couchbase/query/execution.(*base).sendItem+0x217/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/execution/base.go:136
11. 0x7f4e88github.com/couchbase/query/execution.(*PrimaryScan).scanPrimary+0x558/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/execution/scan_primary.go:97
12. 0x808f6agithub.com/couchbase/query/execution.func·036+0xea/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/execution/scan_primary.go:53
13. 0x5408c4sync.(*Once).Do+0xd4/usr/local/go/src/sync/once.go:44
14. 0x7f4916github.com/couchbase/query/execution.(*PrimaryScan).RunOnce+0x66/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/execution/scan_primary.go:54
     
    1 @ 0x419925 0x41bbcc 0x41b062 0x7cb917 0x7f3bd3 0x8081eb 0x5408c4 0x7f3546 0x42d291
15. 0x7cb917github.com/couchbase/query/execution.(*base).sendItem+0x217/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/execution/base.go:136
16. 0x7f3bd3github.com/couchbase/query/execution.(*IntersectScan).sendItems+0xc3/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/execution/scan_intersect.go:172
17. 0x8081ebgithub.com/couchbase/query/execution.func·033+0x4ab/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/execution/scan_intersect.go:126
18. 0x5408c4sync.(*Once).Do+0xd4/usr/local/go/src/sync/once.go:44
19. 0x7f3546github.com/couchbase/query/execution.(*IntersectScan).RunOnce+0x66/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/execution/scan_intersect.go:131
     
    Without any other trace of request around.
    Do you want to have a look?