Details
-
Bug
-
Resolution: Not a Bug
-
Critical
-
7.0.0, 7.1.0
-
Untriaged
-
-
1
-
Unknown
Description
Build: 7.1.0-2021
Test:
Rebalance-in (min) 3->4, 1 bucket x 55M x 1KB, 100 indexes, 10K KV ops/sec, Scan Workload concurrency=128, Plasma 20%RR, s=1 c=10
Test job: http://perf.jenkins.couchbase.com/job/aether/1397/console
Trigger scans and then start to rebalance in the index node
18:36:53 2022-01-12T05:11:35 [INFO] To be applied: ./opt/couchbase/bin/cbindexperf -cluster 172.23.110.72:8091 -auth="Administrator:password" -configfile tests/gsi/plasma/config/config_scanlatency_sessionconsistent_100_indexes_1s_10c_partitioned.json -resultfile result.json -statsfile /root/statsfile -cpuprofile cpuprofile.prof -memprofile memprofile.prof &
|
|
|
18:41:50 2022-01-12T05:16:33 [INFO] Monitoring rebalance status
|
18:41:50 2022-01-12T05:16:35 [INFO] Rebalance progress: 33.33333333333334 %
|
18:41:52 2022-01-12T05:16:37 [INFO] Rebalance progress: 33.33333333333334 %
|
18:41:54 2022-01-12T05:16:39 [INFO] Rebalance progress: 33.33333333333334 %
|
18:41:56 2022-01-12T05:16:41 [INFO] Rebalance progress: 40.00000000000001 %
|
18:41:58 2022-01-12T05:16:43 [INFO] Rebalance progress: 40.00000000000001 %
|
18:42:00 2022-01-12T05:16:45 [INFO] Rebalance progress: 40.00000000000001 %
|
18:42:02 2022-01-12T05:16:47 [INFO] Rebalance progress: 40.00000000000001 %
|
No result file was generated and no errors, I trigger it from node and seeing following errors
root@ubu-sd03:/workspace/workspace/aether/src/perfrunner# ./opt/couchbase/bin/cbindexperf -cluster 172.23.110.72:8091 -auth="Administrator:password" -configfile tests/gsi/plasma/config/config_scanlatency_sessionconsistent_100_indexes_1s_10c_partitioned.json -resultfile result.json -statsfile /root/statsfile -cpuprofile cpuprofile.prof -memprofile memprofile.prof
|
Log Level = error
|
2022/01/12 09:06:48 revrpc: Got error (EOF) and will retry in 1s
|
2022/01/12 09:06:50 revrpc: Got error (EOF) and will retry in 1s
|
2022/01/12 09:06:52 revrpc: Got error (EOF) and will retry in 1s
|
2022/01/12 09:06:54 revrpc: Got error (EOF) and will retry in 1s
|
2022/01/12 09:06:56 revrpc: Got error (EOF) and will retry in 1s
|
2022-01-12T09:06:56.179-08:00 [Error] updateIndexerList(): CBAuth database is stale: last reason: EOF
|
2022/01/12 09:06:58 revrpc: Got error (EOF) and will retry in 1s
|
2022/01/12 09:07:00 revrpc: Got error (EOF) and will retry in 1s
|
2022/01/12 09:07:02 revrpc: Got error (EOF) and will retry in 1s
|
2022-01-12T09:07:02.800-08:00 [Error] [Queryport-connpool:172.23.110.56:9101] doAtuh error in initHostportForAuth: CBAuth database is stale: last reason: EOF
|
2022-01-12T09:07:02.800-08:00 [Error] [Queryport-connpool:172.23.110.56:9101] doAuth error CBAuth database is stale: last reason: EOF in getAuthInfo for connection (172.23.110.74:49960,172.23.110.56:9101)
|
2022-01-12T09:07:02.800-08:00 [Error] Unable to initialize gsi scanclient (172.23.110.56:9101: unable to obtain server version. Error = CBAuth database is stale: last reason: EOF)
|
2022-01-12T09:07:02.801-08:00 [Error] [Queryport-connpool:172.23.110.71:9101] doAtuh error in initHostportForAuth: CBAuth database is stale: last reason: EOF
|
2022-01-12T09:07:02.801-08:00 [Error] [Queryport-connpool:172.23.110.71:9101] doAuth error CBAuth database is stale: last reason: EOF in getAuthInfo for connection (172.23.110.74:44402,172.23.110.71:9101)
|
2022-01-12T09:07:02.801-08:00 [Error] Unable to initialize gsi scanclient (172.23.110.71:9101: unable to obtain server version. Error = CBAuth database is stale: last reason: EOF)
|
Error occured: CBAuth database is stale: last reason: EOF
|