[MB-42825] Join queries hang without returning results - Couchbase database

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: Cheshire-Cat, 6.6.1
Fix Version/s: 6.6.1, 7.0.0
Component/s: query
Labels:
- approved-for-6.6.1
Environment:
Docker for Windows 10 20H2
Image: couchbase/server:7.0.0-beta
All services enabled
Plasma indexes
Loaded beer-sample and travel-sample buckets and created an empty default bucket

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:
CBCollectInfo is attached
Story Points:
1
Is this a Regression?:
Yes

Description

Queries involving joins may hang unexpectedly and never return results. This query in particular, generated by the Linq2Couchbase integration tests, seems to work often:

SELECT `Extent2`.`airportname` as `airportName`, `Extent1`.`airline` as `airline`

FROM `travel-sample` as `Extent1`

INNER JOIN `travel-sample` as `Extent2` ON (`Extent1`.`destinationairport` = `Extent2`.`faa`)

WHERE ((`Extent1`.`type` = 'route') AND (`Extent2`.`type` = 'airport'))

LIMIT 1

I have replicated repeatedly from the .NET SDK, using cURL, and in the Query Workbench. The HTTP request appears to get hung and never return a value and, in the case of cURL, never timeout. However, sometimes (rarely) valid results are returned.

Attachments

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

Attachments

collectinfo-2020-11-19T030751-ns_1@127.0.0.1.zip
13.86 MB
18/Nov/20 7:13 PM

Issue Links

causes

MB-44258 join with double intersect scan returns no rows

Closed

Options
- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

Gerrit Reviews

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

Ascending order - Click to sort in descending order

Brant Burnett created issue - 18/Nov/20 7:14 PM

Hide

Permalink

Sitaram Vemulapalli added a comment - 18/Nov/20 7:54 PM - edited

Repro:
install travel-sample
run the query (couple of times in UI or cbq shell)

Looks like some synchronization issue.

goroutine 184 [sync.Cond.Wait]:

runtime.goparkunlock(...)

        /usr/local/go/src/runtime/proc.go:310

sync.runtime_notifyListWait(0xc001a62648, 0x7fd000000000)

        /usr/local/go/src/runtime/sema.go:510 +0xf8

sync.(*Cond).Wait(0xc001a62638)

        /usr/local/go/src/sync/cond.go:56 +0x9d

github.com/couchbase/query/execution.(*base).baseDone(0xc001a62480)

        /root/master/query/src/github.com/couchbase/query/execution/base.go:307 +0x247

github.com/couchbase/query/execution.(*IntersectScan).Done(0xc001a62480)

        /root/master/query/src/github.com/couchbase/query/execution/scan_intersect.go:257 +0x31

github.com/couchbase/query/execution.(*Sequence).Done(0xc001a62b40)

        /root/master/query/src/github.com/couchbase/query/execution/sequence.go:172 +0xc9

github.com/couchbase/query/execution.(*NLJoin).Done(0xc001a62d80)

        /root/master/query/src/github.com/couchbase/query/execution/join_nl.go:288 +0x6e

github.com/couchbase/query/execution.(*Sequence).Done(0xc001a63200)

        /root/master/query/src/github.com/couchbase/query/execution/sequence.go:172 +0xc9

github.com/couchbase/query/execution.(*Sequence).Done(0xc001a63440)

        /root/master/query/src/github.com/couchbase/query/execution/sequence.go:172 +0xc9

github.com/couchbase/query/execution.(*Authorize).Done(0xc001a63b00)

        /root/master/query/src/github.com/couchbase/query/execution/authorize.go:158 +0x72

github.com/couchbase/query/server.(*BaseRequest).CompleteRequest(0xc001c04000, 0x2ef372d1, 0x2eef6824, 0x1, 0x42, 0x0, 0xc001b60200, 0xc000a4cd20)

Show

Sitaram Vemulapalli added a comment - 18/Nov/20 7:54 PM - edited Repro: install travel-sample run the query (couple of times in UI or cbq shell) Looks like some synchronization issue. goroutine 184 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go: 310 sync.runtime_notifyListWait( 0xc001a62648 , 0x7fd000000000 ) /usr/local/go/src/runtime/sema.go: 510 + 0xf8 sync.(*Cond).Wait( 0xc001a62638 ) /usr/local/go/src/sync/cond.go: 56 + 0x9d github.com/couchbase/query/execution.(*base).baseDone( 0xc001a62480 ) /root/master/query/src/github.com/couchbase/query/execution/base.go: 307 + 0x247 github.com/couchbase/query/execution.(*IntersectScan).Done( 0xc001a62480 ) /root/master/query/src/github.com/couchbase/query/execution/scan_intersect.go: 257 + 0x31 github.com/couchbase/query/execution.(*Sequence).Done( 0xc001a62b40 ) /root/master/query/src/github.com/couchbase/query/execution/sequence.go: 172 + 0xc9 github.com/couchbase/query/execution.(*NLJoin).Done( 0xc001a62d80 ) /root/master/query/src/github.com/couchbase/query/execution/join_nl.go: 288 + 0x6e github.com/couchbase/query/execution.(*Sequence).Done( 0xc001a63200 ) /root/master/query/src/github.com/couchbase/query/execution/sequence.go: 172 + 0xc9 github.com/couchbase/query/execution.(*Sequence).Done( 0xc001a63440 ) /root/master/query/src/github.com/couchbase/query/execution/sequence.go: 172 + 0xc9 github.com/couchbase/query/execution.(*Authorize).Done( 0xc001a63b00 ) /root/master/query/src/github.com/couchbase/query/execution/authorize.go: 158 + 0x72 github.com/couchbase/query/server.(*BaseRequest).CompleteRequest( 0xc001c04000 , 0x2ef372d1 , 0x2eef6824 , 0x1 , 0x42 , 0x0 , 0xc001b60200 , 0xc000a4cd20 )

Sitaram Vemulapalli made changes - 18/Nov/20 7:54 PM

Field	Original Value	New Value
Fix Version/s		Cheshire-Cat [ 15915 ]

Mihir Kamdar made changes - 18/Nov/20 8:35 PM

Affects Version/s

6.6.1 [ 17002 ]

Kamini Jagtiani made changes - 30/Nov/20 9:16 AM

Assignee

Kamini Jagtiani [ kamini.jagtiani ]

Marco Greco [ marco.greco ]

Hide

Permalink

Marco Greco added a comment - 30/Nov/20 10:35 AM

I don't have the complete picture yet, but it seems like the some children of the intersect scan have gone away without notifying the parent

goroutine 91047 [semacquire, 1 minutes]:
sync.runtime_Semacquire(0xc0000e7c54)
/usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc0000e7c54)
/usr/local/go/src/sync/waitgroup.go:130 +0x64
github.com/couchbase/query/execution.(*valueExchange).retrieveChildNoStop(0xc0000e7b80, 0x1)
/home/marco/query/src/github.com/couchbase/query/execution/exchange.go:488 +0xbc
github.com/couchbase/query/execution.(*base).childrenWaitNoStop(0xc0000e7b80, 0x1)
/home/marco/query/src/github.com/couchbase/query/execution/base.go:780 +0x50
github.com/couchbase/query/execution.(*IntersectScan).RunOnce.func1()
/home/marco/query/src/github.com/couchbase/query/execution/scan_intersect.go:151 +0x86c
github.com/couchbase/query/util.(*Once).Do(0xc0000e7cb8, 0xc0019ddf50)
/home/marco/query/src/github.com/couchbase/query/util/sync.go:55 +0x4a
github.com/couchbase/query/execution.(*IntersectScan).RunOnce(0xc0000e7b80, 0xc000e0c000, 0x28b8940, 0xc003b2fe00)
/home/marco/query/src/github.com/couchbase/query/execution/scan_intersect.go:65 +0x82
github.com/couchbase/query/execution.execOp(0x28c08e0, 0xc0000e7b80, 0xc000e0c000, 0x28b8940, 0xc003b2fe00)
/home/marco/query/src/github.com/couchbase/query/execution/base.go:495 +0x54
created by github.com/couchbase/query/execution.(*base).fork
/home/marco/query/src/github.com/couchbase/query/execution/base.go:505 +0xfa

Show

Marco Greco added a comment - 30/Nov/20 10:35 AM I don't have the complete picture yet, but it seems like the some children of the intersect scan have gone away without notifying the parent goroutine 91047 [semacquire, 1 minutes] : sync.runtime_Semacquire(0xc0000e7c54) /usr/local/go/src/runtime/sema.go:56 +0x42 sync.(*WaitGroup).Wait(0xc0000e7c54) /usr/local/go/src/sync/waitgroup.go:130 +0x64 github.com/couchbase/query/execution.(*valueExchange).retrieveChildNoStop(0xc0000e7b80, 0x1) /home/marco/query/src/github.com/couchbase/query/execution/exchange.go:488 +0xbc github.com/couchbase/query/execution.(*base).childrenWaitNoStop(0xc0000e7b80, 0x1) /home/marco/query/src/github.com/couchbase/query/execution/base.go:780 +0x50 github.com/couchbase/query/execution.(*IntersectScan).RunOnce.func1() /home/marco/query/src/github.com/couchbase/query/execution/scan_intersect.go:151 +0x86c github.com/couchbase/query/util.(*Once).Do(0xc0000e7cb8, 0xc0019ddf50) /home/marco/query/src/github.com/couchbase/query/util/sync.go:55 +0x4a github.com/couchbase/query/execution.(*IntersectScan).RunOnce(0xc0000e7b80, 0xc000e0c000, 0x28b8940, 0xc003b2fe00) /home/marco/query/src/github.com/couchbase/query/execution/scan_intersect.go:65 +0x82 github.com/couchbase/query/execution.execOp(0x28c08e0, 0xc0000e7b80, 0xc000e0c000, 0x28b8940, 0xc003b2fe00) /home/marco/query/src/github.com/couchbase/query/execution/base.go:495 +0x54 created by github.com/couchbase/query/execution.(*base).fork /home/marco/query/src/github.com/couchbase/query/execution/base.go:505 +0xfa

Mihir Kamdar made changes - 01/Dec/20 9:57 AM

Labels

approved-for-6.6.1

Mihir Kamdar made changes - 01/Dec/20 9:58 AM

Link

This issue blocks MB-40528 [ MB-40528 ]

Marco Greco made changes - 01/Dec/20 10:02 AM

Fix Version/s

6.6.1 [ 17002 ]

Marco Greco made changes - 01/Dec/20 10:02 AM

Resolution		Fixed [ 1 ]
Status	Open [ 1 ]	Resolved [ 5 ]

Hide

Permalink

Couchbase Build Team added a comment - 01/Dec/20 10:11 AM

Build couchbase-server-7.0.0-3906 contains query commit f40fd5e with commit message:
~~MB-42825~~ hangs on intersect scan

Show

Couchbase Build Team added a comment - 01/Dec/20 10:11 AM Build couchbase-server-7.0.0-3906 contains query commit f40fd5e with commit message: MB-42825 hangs on intersect scan

Hide

Permalink

Couchbase Build Team added a comment - 01/Dec/20 10:27 AM

Build couchbase-server-6.6.1-9200 contains query commit 3c7ab4b with commit message:
~~MB-42825~~ hangs on intersect scan

Show

Couchbase Build Team added a comment - 01/Dec/20 10:27 AM Build couchbase-server-6.6.1-9200 contains query commit 3c7ab4b with commit message: MB-42825 hangs on intersect scan

Wayne Siu made changes - 01/Dec/20 11:56 AM

Assignee

Marco Greco [ marco.greco ]

Mihir Kamdar [ mihir.kamdar ]

Pierre Regazzoni made changes - 01/Dec/20 1:50 PM

Assignee

Mihir Kamdar [ mihir.kamdar ]

Pierre Regazzoni [ JIRAUSER25157 ]

Hide

Permalink

Pierre Regazzoni added a comment - 01/Dec/20 2:14 PM - edited

I could easily repro issue on build 6.6.1-9192.

I then tried on 6.6.1-9200:

I did first run of query with 50 occurrence which worked.
Then tried 100 times but query then hanged on iteration 46.

[root@localhost ~]# cat query-46.out

SELECT `Extent2`.`airportname` as `airportName`, `Extent1`.`airline` as `airline` FROM `travel-sample` as `Extent1` INNER JOIN `travel-sample` as `Extent2` ON (`Extent1`.`destinationairport` = `Extent2`.`faa`) WHERE ((`Extent1`.`type` = 'route') AND (`Extent2`.`type` = 'airport')) LIMIT 1

You can check cluster http://172.23.104.92:8091/ which has 6.6.1-9200.

To repro simply run on server node:

cd /root/

for (( i=0; i<100; ++i)); do /opt/couchbase/bin/cbq -u Administrator -p password -f query.sql -o query-${i}.out; done

Show

Pierre Regazzoni added a comment - 01/Dec/20 2:14 PM - edited I could easily repro issue on build 6.6.1-9192. I then tried on 6.6.1-9200: I did first run of query with 50 occurrence which worked. Then tried 100 times but query then hanged on iteration 46. [root @localhost ~]# cat query- 46 .out SELECT `Extent2`.`airportname` as `airportName`, `Extent1`.`airline` as `airline` FROM `travel-sample` as `Extent1` INNER JOIN `travel-sample` as `Extent2` ON (`Extent1`.`destinationairport` = `Extent2`.`faa`) WHERE ((`Extent1`.`type` = 'route' ) AND (`Extent2`.`type` = 'airport' )) LIMIT 1 You can check cluster http://172.23.104.92:8091/ which has 6.6.1-9200. To repro simply run on server node: cd /root/ for (( i= 0 ; i< 100 ; ++i)); do /opt/couchbase/bin/cbq -u Administrator -p password -f query.sql -o query-${i}.out; done

Pierre Regazzoni made changes - 01/Dec/20 2:14 PM

VERIFICATION STEPS		Run query in repro other 50 times.
Resolution	Fixed [ 1 ]
Status	Resolved [ 5 ]	Reopened [ 4 ]

Pierre Regazzoni made changes - 01/Dec/20 2:15 PM

Assignee

Pierre Regazzoni [ JIRAUSER25157 ]

Marco Greco [ marco.greco ]

Marco Greco made changes - 02/Dec/20 10:49 AM

Resolution		Fixed [ 1 ]
Status	Reopened [ 4 ]	Resolved [ 5 ]

Hide

Permalink

Couchbase Build Team added a comment - 02/Dec/20 11:33 AM

Build couchbase-server-7.0.0-3920 contains query commit 4beab79 with commit message:
~~MB-42825~~ reopen allowed on a stop signal

Show

Couchbase Build Team added a comment - 02/Dec/20 11:33 AM Build couchbase-server-7.0.0-3920 contains query commit 4beab79 with commit message: MB-42825 reopen allowed on a stop signal

Hide

Permalink

Couchbase Build Team added a comment - 02/Dec/20 11:40 AM

Build couchbase-server-6.6.1-9204 contains query commit fe59898 with commit message:
~~MB-42825~~ reopen allowed on a stop signal

Show

Couchbase Build Team added a comment - 02/Dec/20 11:40 AM Build couchbase-server-6.6.1-9204 contains query commit fe59898 with commit message: MB-42825 reopen allowed on a stop signal

Pierre Regazzoni made changes - 02/Dec/20 12:42 PM

Assignee

Marco Greco [ marco.greco ]

Pierre Regazzoni [ JIRAUSER25157 ]

Hide

Permalink

Pierre Regazzoni added a comment - 02/Dec/20 12:58 PM

Verified with following build:

6.6.1-9204
7.0.0-3920
Ran query over 100 times without issue multiple times.
Also did a continuous run for 5 minutes without hangs.

Show

Pierre Regazzoni added a comment - 02/Dec/20 12:58 PM Verified with following build: 6.6.1-9204 7.0.0-3920 Ran query over 100 times without issue multiple times. Also did a continuous run for 5 minutes without hangs.

Pierre Regazzoni made changes - 02/Dec/20 12:58 PM

VERIFICATION STEPS	Run query in repro other 50 times.	Run query in repro other 100 times.
Status	Resolved [ 5 ]	Closed [ 6 ]

Marco Greco made changes - 03/Dec/20 6:28 AM

Assignee	Pierre Regazzoni [ JIRAUSER25157 ]	Marco Greco [ marco.greco ]
Resolution	Fixed [ 1 ]
Status	Closed [ 6 ]	Reopened [ 4 ]

Hide

Permalink

Couchbase Build Team added a comment - 03/Dec/20 9:48 AM

Build couchbase-server-7.0.0-3934 contains query commit ddc4d65 with commit message:
~~MB-42825~~ tighten children waiting

Show

Couchbase Build Team added a comment - 03/Dec/20 9:48 AM Build couchbase-server-7.0.0-3934 contains query commit ddc4d65 with commit message: MB-42825 tighten children waiting

Hide

Permalink

Couchbase Build Team added a comment - 03/Dec/20 9:53 AM

Build couchbase-server-6.6.1-9209 contains query commit b65840a with commit message:
~~MB-42825~~ tighten children waiting

Show

Couchbase Build Team added a comment - 03/Dec/20 9:53 AM Build couchbase-server-6.6.1-9209 contains query commit b65840a with commit message: MB-42825 tighten children waiting

Hide

Permalink

Mihir Kamdar added a comment - 04/Dec/20 7:37 AM

Marco Greco Are we waiting for any further fixes? If not, can this be resolved now ?

Show

Mihir Kamdar added a comment - 04/Dec/20 7:37 AM Marco Greco Are we waiting for any further fixes? If not, can this be resolved now ?

Marco Greco made changes - 04/Dec/20 7:40 AM

Resolution		Fixed [ 1 ]
Status	Reopened [ 4 ]	Resolved [ 5 ]

Wayne Siu made changes - 04/Dec/20 11:15 PM

Assignee

Marco Greco [ marco.greco ]

Mihir Kamdar [ mihir.kamdar ]

Mihir Kamdar made changes - 07/Dec/20 2:03 PM

Assignee

Mihir Kamdar [ mihir.kamdar ]

Pierre Regazzoni [ JIRAUSER25157 ]

Marco Greco made changes - 08/Dec/20 6:56 AM

Status

Resolved [ 5 ]

Closed [ 6 ]

Marco Greco made changes - 11/Feb/21 7:46 AM

Link

This issue causes ~~MB-44258~~ [ ~~MB-44258~~ ]

Lynn Straus made changes - 17/Jun/21 3:22 PM

Fix Version/s

7.0.0 [ 17233 ]

Lynn Straus made changes - 17/Jun/21 3:32 PM

Fix Version/s

Cheshire-Cat [ 15915 ]

People

Assignee:: Pierre Regazzoni

Reporter:: Brant Burnett

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 18/Nov/20 7:14 PM

Updated:: 17/Jun/21 3:32 PM

Resolved:: 04/Dec/20 7:40 AM

Gerrit Reviews

There are no open Gerrit changes

Show There are 8 closed Gerrit changes

Hide There are 8 closed Gerrit changes

MB-42825 hangs on intersect scan: Gerrit Review:

MB-42825 hangs on intersect scan: Gerrit Review:

MB-42825 reopen allowed on a stop signal: Gerrit Review:

MB-42825 tighten children waiting: Gerrit Review:

MB-42825 reopen allowed on a stop signal: Gerrit Review:

MB-42825 tighten children waiting: Gerrit Review:

MB-44258 join with intersect scan returns no rows, regression from MB 42825: Gerrit Review:

MB-44258 join with intersect scan returns no rows, regression from MB 42825: Gerrit Review:

PagerDuty

Couchbase Server

Details

Description

Attachments

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty