Details
Major Description
Install Couchbase server on 4.6.0-3572 on 8 nodes
Create 3 nodes cluster (172.23.106.141, 172.23.107.83, 172.23.106.84) as source
Create default bucket and load 10K items
Create 3 nodes cluster (172.23.121.180, 172.23.121.181, 172.23.107.19) as destination cluster
Create default bucket
At source cluster, set remote cluster (172.23.107.19)
Setting xdcrFailureRestartInterval to 1
Setting xdcrCheckpointInterval to 1800
Starting continuous replication type:xmem from default to default in the remote cluster cluster1 with settings {}
At each node, setting flush param exp_pager_stime to 10 on default
Hard failover 2 nodes at source (141 and 83)
Then add 2 nodes (172.23.108.58, 172.23.108.59) to source cluster.
Wait for 15 seconds.
From remote cluster on node 19, execute cbrecovery
/opt/couchbase/bin/cbrecovery
http://172.23.107.19:8091 http://172.23.106.84:8091
-b default -B default -u Administrator -p password -U Administrator -P password
The execution seem hang and failed with error
| Apparently recovery ns_orchestrator died. Dropped stale recovery status {recovery_status, {running, "default", <<"84687e38327351bad5514ffb09f440f8">>}} | ns_orchestrator 000 | ns_1@172.23.106.84 | 1:19:10 PM Tue Jan 3, 2017 |
Output from test run:
2017-01-03 13:13:22 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.19 2017-01-03 13:13:23 | INFO | MainProcess | Cluster_Thread | [task.execute] command was executed: '/opt/couchbase/bin/cbrecovery http://172.23.107.19:8091 http://172.23.106.84:8091 -b default -B default -u Administrator -p password -U Administrator -P password ' 2017-01-03 13:13:43 | INFO | MainProcess | Cluster_Thread | [task.check] cbrecovery progress: {u'code': u'ok', u'uuid': u'84687e38327351bad5514ffb09f440f8', u'recoveryMap': [{u'node': u'ns_1@172.23.108.58', u'vbuckets': [212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853]}]} 2017-01-03 13:14:03 | INFO | MainProcess | Cluster_Thread | [task.check] cbrecovery progress: {u'code': u'ok', u'uuid': u'84687e38327351bad5514ffb09f440f8', u'recoveryMap': [{u'node': u'ns_1@172.23.108.58', u'vbuckets': [259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853]}]} 2017-01-03 13:14:24 | INFO | MainProcess | Cluster_Thread | [task.check] cbrecovery progress: {u'code': u'ok', u'uuid': u'84687e38327351bad5514ffb09f440f8', u'recoveryMap': [{u'node': u'ns_1@172.23.108.58', u'vbuckets': [305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853]}]} 2017-01-03 13:14:44 | INFO | MainProcess | Cluster_Thread | [task.check] cbrecovery progress: {u'code': u'ok', u'uuid': u'84687e38327351bad5514ffb09f440f8', u'recoveryMap': [{u'node': u'ns_1@172.23.108.58', u'vbuckets': [692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853]}]} 2017-01-03 13:15:04 | INFO | MainProcess | Cluster_Thread | [task.check] cbrecovery progress: {u'code': u'ok', u'uuid': u'84687e38327351bad5514ffb09f440f8', u'recoveryMap': [{u'node': u'ns_1@172.23.108.58', u'vbuckets': [732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853]}]} 2017-01-03 13:15:24 | INFO | MainProcess | Cluster_Thread | [task.check] cbrecovery progress: {u'code': u'ok', u'uuid': u'84687e38327351bad5514ffb09f440f8', u'recoveryMap': [{u'node': u'ns_1@172.23.108.58', u'vbuckets': [772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853]}]} 2017-01-03 13:15:44 | INFO | MainProcess | Cluster_Thread | [task.check] cbrecovery progress: {u'code': u'ok', u'uuid': u'84687e38327351bad5514ffb09f440f8', u'recoveryMap': [{u'node': u'ns_1@172.23.108.58', u'vbuckets': [811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853]}]} 2017-01-03 13:16:04 | INFO | MainProcess | Cluster_Thread | [task.check] cbrecovery progress: {u'code': u'ok', u'uuid': u'84687e38327351bad5514ffb09f440f8', u'recoveryMap': [{u'node': u'ns_1@172.23.108.58', u'vbuckets': [852, 853]}]} 2017-01-03 13:16:24 | WARNING | MainProcess | Cluster_Thread | [task.check] cbrecovery progress was not changed 2017-01-03 13:16:44 | WARNING | MainProcess | Cluster_Thread | [task.check] cbrecovery progress was not changed 2017-01-03 13:17:04 | WARNING | MainProcess | Cluster_Thread | [task.check] cbrecovery progress was not changed 2017-01-03 13:17:24 | WARNING | MainProcess | Cluster_Thread | [task.check] cbrecovery progress was not changed |
Test to run:
cbRecoverytests.cbrecovery.cbrecover_multiple_failover_swapout_reb_routine,items=10000,rdirection=unidirection,ctopology=chain,failover=source,fail_count=2,add_count=2,max_verify=10000
Test was passed in build 4.6.0-3527