Details
-
Bug
-
Resolution: Fixed
-
Test Blocker
-
7.1.0
-
Untriaged
-
-
1
-
Yes
Description
Build : 7.1.0-1742
Test :
Scale : 3
Iteration : 1
In the longevity system test, at the beginning of the iteration, eventing handlers are created and then there is a wait until they are deployed. All the handlers created are stuck in Deploying state for almost 2 hrs now.
[2021-11-18T14:18:44-08:00, sequoiatools/eventing:7.0:85e6ff] eventing_helper.py -i 172.23.104.137 -u Administrator -p password -o deploy
|
[2021-11-18T14:18:49-08:00, sequoiatools/eventing:7.0:33cf61] eventing_helper.py -i 172.23.104.137 -u Administrator -p password -o wait_for_state --state deployed
|
The eventing logs are filled with errors like these :
2021-11-18T16:18:06.400-08:00 [Error] Producer::getOpCallback [timers_0:1] Bucket get failed for key: <ud>eventing::1240154941::timers_0::vb::0</ud> , err: invalid argument | {"status_code":4,"bucket":"bucket3","error_name":"EINVAL","error_description":"Invalid packet","opaque":54630,"context":"Request must not include key","last_dispatched_to":"172.23.104.5:11210","last_dispatched_from":"172.23.121.3:36564","last_connection_id":"f039f9e74dea477d/21ff6ddbdb166186"}
|
2021-11-18T16:18:06.401-08:00 [Error] Producer::getOpCallback [sbm_0:1] Bucket get failed for key: <ud>eventing::4242142605::sbm_0::vb::0</ud> , err: invalid argument | {"status_code":4,"bucket":"bucket3","error_name":"EINVAL","error_description":"Invalid packet","opaque":54633,"context":"Request must not include key","last_dispatched_to":"172.23.104.5:11210","last_dispatched_from":"172.23.121.3:36564","last_connection_id":"f039f9e74dea477d/21ff6ddbdb166186"}
|
2021-11-18T16:18:06.401-08:00 [Error] Producer::getOpCallback [n1ql_0:1] Bucket get failed for key: <ud>eventing::226400845::n1ql_0::vb::0</ud> , err: invalid argument | {"status_code":4,"bucket":"bucket3","error_name":"EINVAL","error_description":"Invalid packet","opaque":54631,"context":"Request must not include key","last_dispatched_to":"172.23.104.5:11210","last_dispatched_from":"172.23.121.3:36564","last_connection_id":"f039f9e74dea477d/21ff6ddbdb166186"}
|
2021-11-18T16:18:06.401-08:00 [Error] Producer::getOpCallback [curl_0:1] Bucket get failed for key: <ud>eventing::3183049456::curl_0::vb::0</ud> , err: invalid argument | {"status_code":4,"bucket":"bucket3","error_name":"EINVAL","error_description":"Invalid packet","opaque":54632,"context":"Request must not include key","last_dispatched_to":"172.23.104.5:11210","last_dispatched_from":"172.23.121.3:36564","last_connection_id":"f039f9e74dea477d/21ff6ddbdb166186"}
|
2021-11-18T16:18:06.401-08:00 [Error] Producer::getOpCallback [curl_0:1] Bucket get failed for key: <ud>eventing::3183049456::curl_0::vb::0</ud> , err: invalid argument | {"status_code":4,"bucket":"bucket3","error_name":"EINVAL","error_description":"Invalid packet","opaque":54632,"context":"Request must not include key","last_dispatched_to":"172.23.104.5:11210","last_dispatched_from":"172.23.121.3:36564","last_connection_id":"f039f9e74dea477d/21ff6ddbdb166186"}
|
2021-11-18T16:18:06.571-08:00 [Error] Consumer::getOpCallback [worker_timers_0_0:/tmp/127.0.0.1:8091_0_1240154941.sock:42866] Bucket fetch failed for key: <ud>eventing::1240154941::timers_0::vb::0</ud>, err: invalid argument | {"status_code":4,"bucket":"bucket3","error_name":"EINVAL","error_description":"Invalid packet","opaque":54634,"context":"Request must not include key","last_dispatched_to":"172.23.104.5:11210","last_dispatched_from":"172.23.121.3:36564","last_connection_id":"f039f9e74dea477d/21ff6ddbdb166186"}
|
2021-11-18T16:18:06.572-08:00 [Error] Consumer::getOpCallback [worker_curl_0_0:/tmp/127.0.0.1:8091_0_3183049456.sock:42865] Bucket fetch failed for key: <ud>eventing::3183049456::curl_0::vb::0</ud>, err: invalid argument | {"status_code":4,"bucket":"bucket3","error_name":"EINVAL","error_description":"Invalid packet","opaque":54635,"context":"Request must not include key","last_dispatched_to":"172.23.104.5:11210","last_dispatched_from":"172.23.121.3:36564","last_connection_id":"f039f9e74dea477d/21ff6ddbdb166186"}
|
2021-11-18T16:18:06.872-08:00 [Error] Consumer::getOpCallback [worker_sbm_0_0:/tmp/127.0.0.1:8091_0_4242142605.sock:42898] Bucket fetch failed for key: <ud>eventing::4242142605::sbm_0::vb::0</ud>, err: invalid argument | {"status_code":4,"bucket":"bucket3","error_name":"EINVAL","error_description":"Invalid packet","opaque":54636,"context":"Request must not include key","last_dispatched_to":"172.23.104.5:11210","last_dispatched_from":"172.23.121.3:36564","last_connection_id":"f039f9e74dea477d/21ff6ddbdb166186"}
|
2021-11-18T16:18:07.176-08:00 [Error] Consumer::getOpCallback [worker_n1ql_0_0:/tmp/127.0.0.1:8091_0_226400845.sock:42847] Bucket fetch failed for key: <ud>eventing::226400845::n1ql_0::vb::0</ud>, err: invalid argument | {"status_code":4,"bucket":"bucket3","error_name":"EINVAL","error_description":"Invalid packet","opaque":54637,"context":"Request must not include key","last_dispatched_to":"172.23.104.5:11210","last_dispatched_from":"172.23.121.3:36564","last_connection_id":"f039f9e74dea477d/21ff6ddbdb166186"}
|
2021-11-18T16:18:07.401-08:00 [Error] Producer::getOpCallback [timers_0:1] Bucket get failed for key: <ud>eventing::1240154941::timers_0::vb::0</ud> , err: invalid argument | {"status_code":4,"bucket":"bucket3","error_name":"EINVAL","error_description":"Invalid packet","opaque":54638,"context":"Request must not include key","last_dispatched_to":"172.23.104.5:11210","last_dispatched_from":"172.23.121.3:36564","last_connection_id":"f039f9e74dea477d/21ff6ddbdb166186"}
|
Prima facie this issue doesn't look like MB-49547.
Logs attached. Eventing nodes : 172.23.104.137, 172.23.121.3, 172.23.123.27
This is a regression, since this issue wasn't seen in any of the recent longevity test runs (Checked on the runs with 7.1.0-1701, 7.1.0-1717, 7.1.0-1727, 7.1.0-1733)
Attachments
Issue Links
- relates to
-
-
- Closed
-
| For Gerrit Dashboard: MB-49666 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 166120,4 | MB-49666: Upgrade gocb and gocbcore | master | manifest | Status: MERGED | +2 | +1 |