Loading...

Details

Type: Bug
Resolution: Duplicate
Priority: Test Blocker
Fix Version/s: 4.0.0
Affects Version/s: 4.0.0
Component/s: query
Security Level: Public
Labels:
- system-test
Environment:

Hide
400-2093
View_indexes -10
2 gsi indexes on default bucket. See attached screenshot for gsi indexes.
6 nodes, ( 2 nodes kv+indexer+query and 4 are KV only)

- Tpcc workload 10M on 9 buckets
- sabre workload on default bucket

Show
400-2093 View_indexes -10 2 gsi indexes on default bucket. See attached screenshot for gsi indexes. 6 nodes, ( 2 nodes kv+indexer+query and 4 are KV only) - Tpcc workload 10M on 9 buckets - sabre workload on default bucket

Triage:
Untriaged
Is this a Regression?:
Unknown

Description

1. create cluster, create view indexes for almost all buckets
2. load tpcc 100 warehouses, 10M items on ORDER_LINE bucekt, attached screenshot for item and server distribution
3. Wait for indexes to be built, steady

4. Start sample sabre querying - ( attached standalone scrpt)

RunOnceQuery - collects all the keys, ids etc for further inputs of transactonal queries
RunNQuery - issues queries with typical expected response of < 10 rows

Following indexes are defined on the system

cbq> select * from system:indexes;
ERROR : Post http://localhost:8093/query: EOF
cbq> select * from system:indexes;
ERROR : Post http://localhost:8093/query: EOF

and other query node shows
cbq> select * from system:indexes;
{
"requestID": "ae343ba8-9f3b-4f11-b347-a25200058563",
"signature":

{ "*": "*" }

,
"results": [
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "fs_origin", "index_key": [ "(((((((`AirItinerary`.`OriginDestinationOptions`).`OriginDestinationOption`)[0]).`FlightSegment`)[0]).`DepatureAirport`).`LocationCode`)" ], "keyspace_id": "default", "name": "fs_origin", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "WH_ID", "index_key": [ "`W_ID`" ], "keyspace_id": "WAREHOUSE", "name": "WH_ID", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "PX_CUSTOMER", "index_key": [ "(meta(`CUSTOMER`).`id`)" ], "is_primary": true, "keyspace_id": "CUSTOMER", "name": "PX_CUSTOMER", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "OR_O_ID_D_ID_W_ID", "index_key": [ "`O_ID`", "`O_D_ID`", "`O_W_ID`", "`O_C_ID`" ], "keyspace_id": "ORDERS", "name": "OR_O_ID_D_ID_W_ID", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "PX_ORDER_LINE", "index_key": [ "(meta(`ORDER_LINE`).`id`)" ], "is_primary": true, "keyspace_id": "ORDER_LINE", "name": "PX_ORDER_LINE", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "e5158142c7c22c47", "index_key": [ "(((`AirItineraryPricingInfo`.`ItinTotalFare`).`TotalFare`).`Amount`)" ], "keyspace_id": "default", "name": "amt", "namespace_id": "default", "state": "online", "using": "gsi" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "OR_W_ID_D_ID_C_ID", "index_key": [ "`O_W_ID`", "`O_D_ID`", "`O_C_ID`" ], "keyspace_id": "ORDERS", "name": "OR_W_ID_D_ID_C_ID", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "IT_ID", "index_key": [ "`I_ID`" ], "keyspace_id": "ITEM", "name": "IT_ID", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "seq_num", "index_key": [ "`SequenceNumber`" ], "keyspace_id": "default", "name": "seq_num", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "NO_D_ID_W_ID", "index_key": [ "`NO_O_ID`", "`NO_D_ID`", "`NO_W_ID`" ], "keyspace_id": "NEW_ORDER", "name": "NO_D_ID_W_ID", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "PX_ORDERS", "index_key": [ "(meta(`ORDERS`).`id`)" ], "is_primary": true, "keyspace_id": "ORDERS", "name": "PX_ORDERS", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "PX_STOCK", "index_key": [ "(meta(`STOCK`).`id`)" ], "is_primary": true, "keyspace_id": "STOCK", "name": "PX_STOCK", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "PX_DISTRICT", "index_key": [ "(meta(`DISTRICT`).`id`)" ], "is_primary": true, "keyspace_id": "DISTRICT", "name": "PX_DISTRICT", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "DI_ID_W_ID", "index_key": [ "`D_ID`", "`D_W_ID`" ], "keyspace_id": "DISTRICT", "name": "DI_ID_W_ID", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "ST_W_ID_I_ID1", "index_key": [ "`S_W_ID`", "`S_I_ID`" ], "keyspace_id": "STOCK", "name": "ST_W_ID_I_ID1", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "px_history", "index_key": [ "(meta(`HISTORY`).`id`)" ], "is_primary": true, "keyspace_id": "HISTORY", "name": "px_history", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "OL_O_ID_D_ID_W_ID", "index_key": [ "`OL_O_ID`", "`OL_D_ID`", "`OL_W_ID`" ], "keyspace_id": "ORDER_LINE", "name": "OL_O_ID_D_ID_W_ID", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "CU_ID_D_ID_W_ID", "index_key": [ "`C_ID`", "`C_D_ID`", "`C_W_ID`" ], "keyspace_id": "CUSTOMER", "name": "CU_ID_D_ID_W_ID", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "px_default", "index_key": [ "(meta(`default`).`id`)" ], "is_primary": true, "keyspace_id": "default", "name": "px_default", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "PX_ITEM", "index_key": [ "(meta(`ITEM`).`id`)" ], "is_primary": true, "keyspace_id": "ITEM", "name": "PX_ITEM", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "4129c421e72619bf", "index_key": [ "(((((((`AirItinerary`.`OriginDestinationOptions`).`OriginDestinationOption`)[0]).`FlightSegment`)[0]).`ArrivalAirport`).`LocationCode`)" ], "keyspace_id": "default", "name": "fs_origin_arriv", "namespace_id": "default", "state": "online", "using": "gsi" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "PX_NEW_ORDER", "index_key": [ "(meta(`NEW_ORDER`).`id`)" ], "is_primary": true, "keyspace_id": "NEW_ORDER", "name": "PX_NEW_ORDER", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "CU_W_ID_D_ID_LAST", "index_key": [ "`C_W_ID`", "`C_D_ID`", "`C_LAST`" ], "keyspace_id": "CUSTOMER", "name": "CU_W_ID_D_ID_LAST", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "d_sequencenumber", "index_key": [ "`SequenceNumber`" ], "keyspace_id": "default", "name": "d_sequencenumber", "namespace_id": "default", "state": "online", "using": "view" }

},
{
"indexes":

{ "datastore_id": "http://127.0.0.1:8091", "id": "PX_WAREHOUSE", "index_key": [ "(meta(`WAREHOUSE`).`id`)" ], "is_primary": true, "keyspace_id": "WAREHOUSE", "name": "PX_WAREHOUSE", "namespace_id": "default", "state": "online", "using": "view" }

}
],
"status": "success",
"metrics":

{ "elapsedTime": "5.627979532s", "executionTime": "5.627746696s", "resultCount": 25, "resultSize": 11740 }

}

I started seeing too many crashes currently on query.logs with data from TPCC and sabre together.
This could be same/related to ~~MB-15027~~

[root@centos-64-x64 logs]# grep -r "panic" query.log
panic: Invalid opaque Value. Debug info : Res.opaque : 1, Keys 1, Response received MCResponse status=SUCCESS, opcode=GET, opaque=1, msg:

{"NO_D_ID":10,"NO_O_ID":0,"NO_W_ID":81}

runtime.gopanic(0xb1c500, 0xc209447c00)
/usr/local/go/src/runtime/panic.go:425 +0x2a3 fp=0xc20ea07e10 sp=0xc20ea07da8
panic: Invalid opaque Value. Debug info : Res.opaque : 1, Keys 1, Response received MCResponse status=SUCCESS, opcode=GET, opaque=1, msg:

{"NO_D_ID":3,"NO_O_ID":3001,"NO_W_ID":26}

runtime.gopanic(0xb1c500, 0xc225e729b0)
/usr/local/go/src/runtime/panic.go:425 +0x2a3 fp=0xc20bfc9e10 sp=0xc20bfc9da8
panic: Invalid opaque Value. Debug info : Res.opaque : 1, Keys 1, Response received MCResponse status=SUCCESS, opcode=GET, opaque=1, msg:

{"NO_D_ID":7,"NO_O_ID":0,"NO_W_ID":26}

runtime.gopanic(0xb1c500, 0xc2288294f0)
/usr/local/go/src/runtime/panic.go:425 +0x2a3 fp=0xc22b029e10 sp=0xc22b029da8
panic: Invalid opaque Value. Debug info : Res.opaque : 1, Keys 1, Response received MCResponse status=SUCCESS, opcode=GET, opaque=1, msg:

{"NO_D_ID":7,"NO_O_ID":0,"NO_W_ID":57}

runtime.gopanic(0xb1c500, 0xc2138fddb0)
/usr/local/go/src/runtime/panic.go:425 +0x2a3 fp=0xc210c49e10 sp=0xc210c49da8
panic: Invalid opaque Value. Debug info : Res.opaque : 1, Keys 1, Response received MCResponse status=SUCCESS, opcode=GET, opaque=1, msg:

{"NO_D_ID":2,"NO_O_ID":3003,"NO_W_ID":93}

runtime.gopanic(0xb1c500, 0xc2164c6d30)
/usr/local/go/src/runtime/panic.go:425 +0x2a3 fp=0xc21b55be10 sp=0xc21b55bda8
panic: Invalid opaque Value. Debug info : Res.opaque : 1, Keys 1, Response received MCResponse status=SUCCESS, opcode=GET, opaque=1, msg:

{"NO_D_ID":1,"NO_O_ID":2101,"NO_W_ID":26}

runtime.gopanic(0xb1c500, 0xc20b003c70)
/usr/local/go/src/runtime/panic.go:425 +0x2a3 fp=0xc20b823e10 sp=0xc20b823da8
panic: Invalid opaque Value. Debug info : Res.opaque : 1, Keys 1, Response received MCResponse status=SUCCESS, opcode=GET, opaque=1, msg:

{"NO_D_ID":5,"NO_O_ID":0,"NO_W_ID":88}

runtime.gopanic(0xb1c500, 0xc2239bdc30)
/usr/local/go/src/runtime/panic.go:425 +0x2a3 fp=0xc214701e10 sp=0xc214701da8
panic: Invalid opaque Value. Debug info : Res.opaque : 1, Keys 1, Response received MCResponse status=SUCCESS, opcode=GETQ, opaque=1, msg:

{"NO_D_ID":7,"NO_O_ID":0,"NO_W_ID":49}

runtime.gopanic(0xb1c500, 0xc21acb00e0)
/usr/local/go/src/runtime/panic.go:425 +0x2a3 fp=0xc209283e10 sp=0xc209283da8
panic: Invalid opaque Value. Debug info : Res.opaque : 1, Keys 1, Response received MCResponse status=SUCCESS, opcode=GET, opaque=1, msg:

{"NO_D_ID":2,"NO_O_ID":3001,"NO_W_ID":67}

runtime.gopanic(0xb1c500, 0xc2248d8f90)
/usr/local/go/src/runtime/panic.go:425 +0x2a3 fp=0xc22844be10 sp=0xc22844bda8
panic: Invalid opaque Value. Debug info : Res.opaque : 1, Keys 1, Response received MCResponse status=SUCCESS, opcode=GET, opaque=1, msg:

{"NO_D_ID":7,"NO_O_ID":0,"NO_W_ID":88}

runtime.gopanic(0xb1c500, 0xc217d17ee0)
/usr/local/go/src/runtime/panic.go:425 +0x2a3 fp=0xc213b5be10 sp=0xc213b5bda8