Mahesh Mandhare, Looks like this issue would have been fixed/improved by the change set for MB-34911.
Can we get this retested on build 6.5.0-4522 or the latest.
Sreekanth Sivasankaran
added a comment - Mahesh Mandhare , Looks like this issue would have been fixed/improved by the change set for MB-34911 .
Can we get this retested on build 6.5.0-4522 or the latest.
Given that a 2 node rebalance in/out operation for 30 indexes, wouldn't be surprised that it took 10 hours. But this needs to be investigated further. Mostly for CC as we don't have any immediate fixes available for this.
Sreekanth Sivasankaran
added a comment - Good to to know that the rebalance passed now.
Given that a 2 node rebalance in/out operation for 30 indexes, wouldn't be surprised that it took 10 hours. But this needs to be investigated further. Mostly for CC as we don't have any immediate fixes available for this.
Mahesh Mandhare (Inactive)
added a comment - Build 6.5.0-4959
On this build rebalance out of FTS node took 1739.46 min.
Job- http://perf.jenkins.couchbase.com/job/arke-multi-bucket/345
Shivani Gupta, unlike the ticket title says - this rebalance passes now, just that it progresses really slow. The scaling fix for this needs to be a really thought through and a comprehensive one. We are not planning any performance improvement fixes in 6.5.1.
Sreekanth Sivasankaran
added a comment - Shivani Gupta , unlike the ticket title says - this rebalance passes now, just that it progresses really slow. The scaling fix for this needs to be a really thought through and a comprehensive one. We are not planning any performance improvement fixes in 6.5.1.
Sreekanth Sivasankaran thanks. Can you please fix the title so that it does not cause confusion going forward?
Shivani Gupta (Inactive)
added a comment - Sreekanth Sivasankaran thanks. Can you please fix the title so that it does not cause confusion going forward?
"Given that a 2 node rebalance in/out operation for 30 indexes, wouldn't be surprised that it took 10 hours. But this needs to be investigated further. Mostly for CC as we don't have any immediate fixes available for this."
This is not expected behavior. As mentioned by Mahesh Mandhare on 6.5.0-2082, this rebalance out took 79 min.
Clearly there is some regression here. Please see if this can be addressed for 6.5.1.
Shivani Gupta (Inactive)
added a comment - - edited Sreekanth Sivasankaran in response to your comment:
"Given that a 2 node rebalance in/out operation for 30 indexes, wouldn't be surprised that it took 10 hours. But this needs to be investigated further. Mostly for CC as we don't have any immediate fixes available for this."
This is not expected behavior. As mentioned by Mahesh Mandhare on 6.5.0-2082, this rebalance out took 79 min.
Clearly there is some regression here. Please see if this can be addressed for 6.5.1.
Keshav Murthy FYI.
Thanks Shivani Gupta for brining this point right to my radar, I went completely oblivious(missed that point in description) to the fact that It used to work in older builds. Let me check this asap.
Sreekanth Sivasankaran
added a comment - Thanks Shivani Gupta for brining this point right to my radar, I went completely oblivious(missed that point in description) to the fact that It used to work in older builds. Let me check this asap.
In the recent logs of the rebalance out operation, I can see heavy app_herder intervention (345 job run) to slow down things. Working with Mahesh Mandhare to get the older logs to verify/debug how it used to work earlier without the memory pressure.
Per review in the Cheshire Cat Ops meeting and per email updates, all remaining majors/minors/trivials are being moved out of Cheshire Cat on March 10. The purpose is to allow teams to focus on Critical bugs for the remainder of the release. Teams can "pull back" deferred bugs as schedule and bandwidth allow.
Changing target fix version to CheshireCat.next and adding "deferred-from-Cheshire-Cat" label.
Lynn Straus
added a comment - Per review in the Cheshire Cat Ops meeting and per email updates, all remaining majors/minors/trivials are being moved out of Cheshire Cat on March 10. The purpose is to allow teams to focus on Critical bugs for the remainder of the release. Teams can "pull back" deferred bugs as schedule and bandwidth allow.
Changing target fix version to CheshireCat.next and adding "deferred-from-Cheshire-Cat" label.
Jyotsna Nayak
added a comment - Have rerun this test on CC build: 7.0.0-4554.
Out rebalance completed. Time taken for swap rebalance : 78.15 min ( 1hr 18 mins)
Link to the job: http://perf.jenkins.couchbase.com/job/themis_multibucket/63/consoleFull
Mahesh Mandhare, Looks like this issue would have been fixed/improved by the change set for
MB-34911.Can we get this retested on build 6.5.0-4522 or the latest.