Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

[BP to 7.0.5 MB-54195] turn off impersonation if the KV node does not support it

Description

After an upgrade from 6.6.x to 7.1.x, with auditing turn on requests fail with

com.couchbase.client.core.error.IndexFailureException: The server reported an issue with the underlying index {"completed":true,"coreId":"0xb3ab2cf600000001","errors":[{"code":12008,"message":"Error performing bulk get operation - cause: {4 errors, starting with MCResponse status=0x80, opcode=GET, opaque=2, msg: }","retry":true}],"httpStatus":200,...

Components

Affects versions

Fix versions

Labels

Environment

None

Link to Log File, atop/blg, CBCollectInfo, Core dump

None

Release Notes Description

None

Attachments

2
  • 08 Nov 2022, 01:12 AM
  • 08 Nov 2022, 01:12 AM

Activity

Ajay Pal Bhullar November 10, 2022 at 5:14 PM

closing based on marco's comment, additionally I tried running via cbq and it worked. Seems like it was some problem with my browser caching ui data

Marco Greco November 9, 2022 at 4:10 PM

if you have a look at the audit logs in 173 and 110, you will see the audit log entries above.
FWIW I use the UI.

Ajay Pal Bhullar November 9, 2022 at 4:04 PM
Edited

will try again today, I didnt explicitly set the query context anywhere but maybe the ui just automatically sets it to default:default._default, will try again using cbq instead of the UI and get back to you

Marco Greco November 9, 2022 at 10:52 AM

Error 12021 is unexpected, and it is down to the KV node not understanding collections: you must have had query context set.
Once you unset query_context, or use select * from default:default, you get:

From 173, query audit

{"clientContextId":"01c409a9-4117-476d-af6c-8ec49fb9fff4","description":"A N1QL SELECT statement was executed","errors":null,"id":28672,"isAdHoc":true,"local":{"ip":"172.23.104.173","port":8093},"metrics":{"elapsedTime":"15.762224ms","executionTime":"15.661102ms","resultCount":1,"resultSize":77},"name":"SELECT statement","node":"172.23.104.173:8091","real_userid":{"domain":"local","user":"Administrator"},"remote":{"ip":"172.23.104.173","port":51088},"requestId":"534a07bd-1278-4c90-b5b5-9e957666c96e","statement":"select * from default:default","status":"success","timestamp":"2022-11-09T02:39:36.952-08:00","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0 (Couchbase Query Workbench)"}

From 110, KV audit:

{"bucket":"default","description":"Document was read","id":20488,"key":"<ud>test</ud>","local":{"ip":"172.23.104.110","port":11210},"name":"document read","real_userid":{"domain":"local","user":"@cbq-engine"},"remote":{"ip":"172.23.104.173","port":47726},"timestamp":"2022-11-09T02:45:35.276775-08:00"}

Note how the real user id in the KV audit log is @cbq-engine, which is the result of the fix.

Ajay Pal Bhullar November 8, 2022 at 1:04 AM
Edited

here is the repro i tried

Create a 2 node 6.6.5 cluster, 1 KV, 1 Query. Index is not necessary, but if used, can go on either node (or standalone).
Create a bucket, insert 1 document.
Turn on auditing for query - audit selects.
Select from query, and check that you are auditing correctly.
Upgrade query only to 7.0.5-7642
Select again.
The select shouldn't fail with

{"code":12008,"message":"Error performing bulk get operation - cause: {4 errors, starting with MCResponse status=0x80, opcode=GET, opaque=2, msg: }","retry":true}
And of, course the mixed mode select should be audited correctly.

I do not see the error "error performing bulk get" upon upgrading, but i also do not see the audit entry for the select i executed.

tried select * from default (in mixed mode)

got this error (i assume expected)

[
{
"code": 12021,
"msg": "Scope not found in CB datastore default:default._default",
"query": "select * from default"
}
]

If I check audit.log the specific select statement is not there but it does seem to search for the keyspace on behalf of hte n1ql statement

{"description":"Successful login to couchbase cluster","id":8192,"local":{"ip":"172.23.104.173","port":8091},"name":"login success","real_userid":{"domain":"builtin","user":"Administrator"},"remote":{"ip":"192.168.100.19","port":53684},"roles":["admin"],"sessionid":"b5e6b8d4d7fee0173b6a9853afc7a8fb4f0a23a0","timestamp":"2022-11-07T17:03:16.769-08:00"} {"description":"A N1QL SELECT statement was executed","errors":null,"id":28672,"isAdHoc":true,"local":{"ip":"172.23.104.173","port":8093},"metrics":{"elapsedTime":"29.173914ms","executionTime":"29.03232ms","resultCount":1,"resultSize":18},"name":"SELECT statement","node":"172.23.104.173:8091","real_userid":{"domain":"local","user":"Administrator"},"remote":{"ip":"172.23.104.173","port":36917},"requestId":"646d861d-82f6-468e-a4ce-06221ac2cdd9","statement":"select keyspaces.name from system:keyspaces;","status":"success","timestamp":"2022-11-07T17:03:22.941-08:00","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"} {"description":"A N1QL SELECT statement was executed","errors":null,"id":28672,"isAdHoc":true,"local":{"ip":"172.23.104.173","port":8093},"metrics":{"elapsedTime":"22.507812ms","executionTime":"22.397786ms","resultCount":1,"resultSize":18},"name":"SELECT statement","node":"172.23.104.173:8091","real_userid":{"domain":"local","user":"Administrator"},"remote":{"ip":"172.23.104.173","port":36917},"requestId":"23ecc841-b4bd-4cfc-842d-5ece2348a77f","statement":"select keyspaces.name from system:keyspaces;","status":"success","timestamp":"2022-11-07T17:03:23.344-08:00","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"}

logs attached, .110 is kv node, .173 is query node

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Is this a Regression?

Unknown

Triage

Untriaged

Story Points

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created October 28, 2022 at 7:19 PM
Updated November 10, 2022 at 5:14 PM
Resolved November 10, 2022 at 5:14 PM
Instabug