observed several crashes in perf test Rebalance-in (min), 4->5 -> 5->5 UniDir, 1 bucket x 1G x 1KB, 10K ops/sec

Test: tests/reb_in_1G_dgm_xdcr_45_55_unidir.test
Build: 4.0.0-3359

http://ci.sc.couchbase.com/job/xdcr-5x5/935

1.
panic: Invalid opcode, 0xa2, in DataFailedCRSource event from xmem_1848b85cd668f659f35092f8ab751d0e/bucket-1/bucket-1_172.23.96.106:11210_2.

goroutine 73169 [running]:
github.com/couchbase/goxdcr/pipeline_svc.(*outNozzleCollector).ProcessEvent(0xc25e0c4510, 0xc20d905b80, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/statistics_manager.go:886 +0x13e1
github.com/couchbase/goxdcr/component.(*AsyncComponentEventListenerImpl).start(0xc2480f9d10)
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/goxdcr/component/async_listener.go:74 +0x1cd
created by github.com/couchbase/goxdcr/component.(*AsyncComponentEventListenerImpl).Start
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/goxdcr/component/async_listener.go:57 +0x54

2.
XmemNozzle 2015-07-10T20:46:19.180-07:00 [INFO] xmem_2d2e2aef07f5a307a4b486cb718ffaee/bucket-1/bucket-1_172.23.96.108:11210_18 is not running, no need to repairConn
XmemNozzle 2015-07-10T20:46:19.180-07:00 [INFO] xmem_2d2e2aef07f5a307a4b486cb718ffaee/bucket-1/bucket-1_172.23.96.108:11210_18 processData_batch exits
XmemNozzle 2015-07-10T20:46:19.180-07:00 [INFO] xmem_2d2e2aef07f5a307a4b486cb718ffaee/bucket-1/bucket-1_172.23.96.108:11210_16 is not running, no need to repairConn
XmemNozzle 2015-07-10T20:46:19.180-07:00 [ERROR] The connection is ruined. Repair the connection and retry.
XmemNozzle 2015-07-10T20:46:19.180-07:00 [INFO] xmem_2d2e2aef07f5a307a4b486cb718ffaee/bucket-1/bucket-1_172.23.96.108:11210_16 is not running, no need to repairConn
XmemNozzle 2015-07-10T20:46:19.180-07:00 [INFO] xmem_2d2e2aef07f5a307a4b486cb718ffaee/bucket-1/bucket-1_172.23.96.108:11210_16 receiveResponse exits
panic: method addSentSeqno in tracker service for pipeline 2d2e2aef07f5a307a4b486cb718ffaee/bucket-1/bucket-1_ThroughSeqnoTracker received invalid vbno. vbno=0; valid vbnos=map[739:true 698:true 817:true 718:true 674:true 649:true 775:true 730:true 768:true 690:true 655:true 737:true 693:true 712:true 780:true 741:true 620:true 728:true 811:true 783:true 750:true 757:true 616:true 755:true 796:true 683:true 818:true 771:true 642:true 749:true 762:true 648:true 814:true 689:true 719:true 716:true 794:true 634:true 678:true 819:true 727:true 807:true 787:true 789:true 640:true 758:true 760:true 636:true 726:true 816:true 709:true 644:true 708:true 667:true 772:true 802:true 629:true 650:true 626:true 809:true 805:true 813:true 782:true 713:true 696:true 731:true 657:true 670:true 808:true 637:true 653:true 788:true 697:true 699:true 795:true 721:true 791:true 806:true 744:true 710:true 769:true 707:true 672:true 722:true 630:true 643:true 645:true 622:true 625:true 715:true 736:true 656:true 747:true 738:true 801:true 673:true 633:true 682:true 765:true 767:true 671:true 675:true 660:true 704:true 681:true 684:true 687:true 799:true 800:true 664:true 668:true 763:true 733:true 786:true 688:true 694:true 679:true 723:true 677:true 646:true 754:true 773:true 703:true 781:true 774:true 711:true 745:true 702:true 753:true 639:true 705:true 740:true 651:true 815:true 685:true 659:true 810:true 627:true 700:true 725:true 812:true 701:true 779:true 777:true 619:true 661:true 803:true 732:true 641:true 748:true 666:true 632:true 790:true 628:true 652:true 770:true 692:true 714:true 618:true 676:true 751:true 759:true 665:true 669:true 662:true 720:true 617:true 631:true 764:true 658:true 743:true 717:true 797:true 804:true 734:true 785:true 706:true 647:true 724:true 695:true 793:true 798:true 742:true 766:true 752:true 615:true 756:true 638:true 735:true 624:true 635:true 792:true 654:true 691:true 663:true 680:true 623:true 784:true 729:true 746:true 686:true 761:true 778:true 776:true 621:true]

goroutine 260110 [running]:
github.com/couchbase/goxdcr/service_impl.(*ThroughSeqnoTrackerSvc).validateVbno(0xc27e091810, 0x0, 0x9e8f70, 0xc)
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/through_seqno_tracker_service.go:420 +0x288
github.com/couchbase/goxdcr/service_impl.(*ThroughSeqnoTrackerSvc).addSentSeqno(0xc27e091810, 0x970000, 0x0)
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/through_seqno_tracker_service.go:191 +0x54
github.com/couchbase/goxdcr/service_impl.(*ThroughSeqnoTrackerSvc).ProcessEvent(0xc27e091810, 0xc264317d10, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/through_seqno_tracker_service.go:167 +0xd5
github.com/couchbase/goxdcr/component.(*AsyncComponentEventListenerImpl).start(0xc2799566e0)
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/goxdcr/component/async_listener.go:74 +0x1cd
created by github.com/couchbase/goxdcr/component.(*AsyncComponentEventListenerImpl).Start
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/goxdcr/component/async_listener.go:57 +0x54