Loading...

Details

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: None
Affects Version/s: 3.2.1
Component/s: library
Labels:
None

Story Points:
1

Description

Build : 7.1.0 build 1444
Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_basic_freetier.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat_free_tier.yml

Cluster with 3 nodes having kv,n1ql, search, index on all the nodes
Create 1 bucket, 100 scopes and 10 collections in each scopes
Create 2500 GSI indexes ( 5 on each collection)
Load documents on some of the collections
Created 1000 indexes: one index (1 partition) on each collection
mutate on one collection and copy all the documents from this collection to other collections

We see below errors in the logs:

memcached.log:2021-10-12T19:23:35.760389+00:00 ERROR 254: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:23:38.761974+00:00 ERROR 256: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:23:41.657465+00:00 ERROR 256: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:23:44.370747+00:00 ERROR 261: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:23:51.119500+00:00 ERROR 262: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:23:59.279735+00:00 ERROR 254: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:24:01.962674+00:00 ERROR 254: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:24:04.642535+00:00 ERROR 262: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:24:07.322540+00:00 ERROR 203: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:24:12.522222+00:00 ERROR 256: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:24:18.946853+00:00 ERROR 261: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:24:21.232128+00:00 ERROR 262: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:24:23.515593+00:00 ERROR 261: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:24:25.815891+00:00 ERROR 203: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

memcached.log:2021-10-12T19:24:30.243935+00:00 ERROR 256: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-09T19:16:13.604Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-09T19:16:13.402446+00:00 ERROR 197: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:23:35.962Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:23:35.760389+00:00 ERROR 254: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:23:38.963Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:23:38.761974+00:00 ERROR 256: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:23:41.859Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:23:41.657465+00:00 ERROR 256: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:23:44.571Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:23:44.370747+00:00 ERROR 261: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:23:51.320Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:23:51.119500+00:00 ERROR 262: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:23:59.480Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:23:59.279735+00:00 ERROR 254: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:24:02.163Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:24:01.962674+00:00 ERROR 254: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:24:04.843Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:24:04.642535+00:00 ERROR 262: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:24:07.523Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:24:07.322540+00:00 ERROR 203: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:24:12.724Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:24:12.522222+00:00 ERROR 256: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:24:19.147Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:24:18.946853+00:00 ERROR 261: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:24:21.433Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:24:21.232128+00:00 ERROR 262: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:24:23.716Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:24:23.515593+00:00 ERROR 261: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:24:26.017Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:24:25.815891+00:00 ERROR 203: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

ns_server.babysitter.log:[ns_server:info,2021-10-12T19:24:30.445Z,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: 2021-10-12T19:24:30.243935+00:00 ERROR 256: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

At this time test was running below queries:

{"log":"2021-10-12 12:21:47,836 - ftsindexmanager - INFO - For upsert into bucket1.scope_76.coll_6 (key _k, value _v) select meta().id _k, _v from bucket1.scope_0.coll_0 _v, Status: QueryStatus.RUNNING, Length of results: 0\n","stream":"stderr","time":"2021-10-12T19:21:47.836940381Z"}

{"log":"2021-10-12 12:21:47,836 - ftsindexmanager - INFO - query: upsert into bucket1.scope_76.coll_5 (key _k, value _v) select meta().id _k, _v from bucket1.scope_0.coll_0 _v\n","stream":"stderr","time":"2021-10-12T19:21:47.837020737Z"}

{"log":"2021-10-12 12:23:54,453 - ftsindexmanager - INFO - For upsert into bucket1.scope_76.coll_5 (key _k, value _v) select meta().id _k, _v from bucket1.scope_0.coll_0 _v, Status: QueryStatus.RUNNING, Length of results: 0\n","stream":"stderr","time":"2021-10-12T19:23:54.454327353Z"}

{"log":"2021-10-12 12:23:54,454 - ftsindexmanager - INFO - query: upsert into bucket1.scope_76.coll_4 (key _k, value _v) select meta().id _k, _v from bucket1.scope_0.coll_0 _v\n","stream":"stderr","time":"2021-10-12T19:23:54.454499154Z"}

{"log":"2021-10-12 12:25:59,575 - ftsindexmanager - INFO - For upsert into bucket1.scope_76.coll_4 (key _k, value _v) select meta().id _k, _v from bucket1.scope_0.coll_0 _v, Status: QueryStatus.RUNNING, Length of results: 0\n","stream":"stderr","time":"2021-10-12T19:25:59.575954051Z"}

{"log":"2021-10-12 12:25:59,575 - ftsindexmanager - INFO - query: upsert into bucket1.scope_76.coll_3 (key _k, value _v) select meta().id _k, _v from bucket1.scope_0.coll_0 _v\n","stream":"stderr","time":"2021-10-12T19:25:59.576039135Z"}

{"log":"2021-10-12 12:28:00,002 - ftsindexmanager - INFO - For upsert into bucket1.scope_76.coll_3 (key _k, value _v) select meta().id _k, _v from bucket1.scope_0.coll_0 _v, Status: QueryStatus.RUNNING, Length of results: 0\n","stream":"stderr","time":"2021-10-12T19:28:00.002419845Z"}

{"log":"2021-10-12 12:28:00,002 - ftsindexmanager - INFO - query: upsert into bucket1.scope_76.coll_2 (key _k, value _v) select meta().id _k, _v from bucket1.scope_0.coll_0 _v\n","stream":"stderr","time":"2021-10-12T19:28:00.0025096Z"}

Logs:

https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-10-13T215808-ns_1%40172.31.50.113.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-10-13T215808-ns_1%40172.31.50.61.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-10-13T215808-ns_1%40172.31.50.69.zip

Attachments

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Closing connection: Connection::isPacketAvailable(): Invalid packet header detected

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty