Build
4.0.0-1869
We noted a peculiar problem on upgrade with xdcr (with encryption) to Sherlock.
-2 clusters - C1 [.11,.16] , C2[.19,.20]
-Both C1 & C2 replicated to each other with encryption.
-Both were offline upgraded to Sherlock in parallel
-After upgrade, C2 failed migration due to old certificate not containing IP sans while C1 passed migration!
Migration on C1.
MigrationService 2015-04-16T20:29:51.825-07:00 [INFO] Starting to migrate remote cluster
MigrationService 2015-04-16T20:29:51.825-07:00 [INFO] data=map[certificate:----BEGIN CERTIFICATE----
MIICmDCCAYKgAwIBAgIIE9WvNtEOgiwwCwYJKoZIhvcNAQEFMAwxCjAIBgNVBAMT
ASowHhcNMTMwMTAxMDAwMDAwWhcNNDkxMjMxMjM1OTU5WjAMMQowCAYDVQQDEwEq
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3ADCSv4CpIrBzkVD0049
TEcbcr9F0NCj3FLo0FLwqikJBOO+psaeccGY32mtjcOEzPtJyt+eykSozDrxWfC/
b6Xx/p1nO4jAKh8u8s5RK4eoioZUcE8KsOBWnkPnwNy4uBtZSHZBlbv19wx4aIIk
Zoe03Hj2blXFZ36SXZEuBNteV8dwBIovB5JNjpSFgdRAMw9LmhEnlkJ1aYImdtj+
ga3S/d/06FuinM+YNioRglANbsvT38kElP5ko+FLvEkbqKH0zyv7yCeV9Sr3++jh
A2Yl0bcLpQX+uKEBkh7h1LXV1/MHam8i3FZUaiCI6XNbyNO6zUywDVvHk3iLSMEH
RQIDAQABowIwADALBgkqhkiG9w0BAQUDggEBAE5QrDQbKiBMg5ZMIeSm5Sa9Xd2t
t97yE2wL/WNpO/2zbW8CkKo3LYXMP3KGI/Wgc7r+ZsqCmByOWi61C02+IqYISopn
S7gVzb2kNOrj5ocCWy5PVFVldtC7H6IOf2b0cgfZfmZIXELoAmKSDtCHo+2quJ+P
56NH4sqU+XblnxajEBVWyIuBxLld17t043n8IGmkD86Ik+GR9JPYJoCXYR6g3RLP
h6h0O5i1Vctv6BC/oDcupnevuYcaOg7HkrcR83dIx4dttwEQHF54+ctfFu45iAFl
MfWlouJfmS6OBVY4NgtteoGNqXVCW9EMzwTqnky49BL3n4pjUijYs5ikMxY=
----END CERTIFICATE----
name:remote_cluster_C1-C2 hostname:10.1.2.19:8091 username:Administrator uuid:b58c5f4927e19f6bee1a426f151f4622 deleted:false password:password demandEncryption:true]
MigrationService 2015-04-16T20:29:51.825-07:00 [INFO] Remote cluster constructed = &
{remoteCluster/b58c5f4927e19f6bee1a426f151f4622 b58c5f4927e19f6bee1a426f151f4622 remote_cluster_C1-C2 10.1.2.19:8091 Administrator password true [45 45 45 45 45 66 69 71 73 78 32 67 69 82 84 73 70 73 67 65 84 69 45 45 45 45 45 10 77 73 73 67 109 68 67 67 65 89 75 103 65 119 73 66 65 103 73 73 69 57 87 118 78 116 69 79 103 105 119 119 67 119 89 74 75 111 90 73 104 118 99 78 65 81 69 70 77 65 119 120 67 106 65 73 66 103 78 86 66 65 77 84 10 65 83 111 119 72 104 99 78 77 84 77 119 77 84 65 120 77 68 65 119 77 68 65 119 87 104 99 78 78 68 107 120 77 106 77 120 77 106 77 49 79 84 85 53 87 106 65 77 77 81 111 119 67 65 89 68 86 81 81 68 69 119 69 113 10 77 73 73 66 73 106 65 78 66 103 107 113 104 107 105 71 57 119 48 66 65 81 69 70 65 65 79 67 65 81 56 65 77 73 73 66 67 103 75 67 65 81 69 65 51 65 68 67 83 118 52 67 112 73 114 66 122 107 86 68 48 48 52 57 10 84 69 99 98 99 114 57 70 48 78 67 106 51 70 76 111 48 70 76 119 113 105 107 74 66 79 79 43 112 115 97 101 99 99 71 89 51 50 109 116 106 99 79 69 122 80 116 74 121 116 43 101 121 107 83 111 122 68 114 120 87 102 67 47 10 98 54 88 120 47 112 49 110 79 52 106 65 75 104 56 117 56 115 53 82 75 52 101 111 105 111 90 85 99 69 56 75 115 79 66 87 110 107 80 110 119 78 121 52 117 66 116 90 83 72 90 66 108 98 118 49 57 119 120 52 97 73 73 107 10 90 111 101 48 51 72 106 50 98 108 88 70 90 51 54 83 88 90 69 117 66 78 116 101 86 56 100 119 66 73 111 118 66 53 74 78 106 112 83 70 103 100 82 65 77 119 57 76 109 104 69 110 108 107 74 49 97 89 73 109 100 116 106 43 10 103 97 51 83 47 100 47 48 54 70 117 105 110 77 43 89 78 105 111 82 103 108 65 78 98 115 118 84 51 56 107 69 108 80 53 107 111 43 70 76 118 69 107 98 113 75 72 48 122 121 118 55 121 67 101 86 57 83 114 51 43 43 106 104 10 65 50 89 108 48 98 99 76 112 81 88 43 117 75 69 66 107 104 55 104 49 76 88 86 49 47 77 72 97 109 56 105 51 70 90 85 97 105 67 73 54 88 78 98 121 78 79 54 122 85 121 119 68 86 118 72 107 51 105 76 83 77 69 72 10 82 81 73 68 65 81 65 66 111 119 73 119 65 68 65 76 66 103 107 113 104 107 105 71 57 119 48 66 65 81 85 68 103 103 69 66 65 69 53 81 114 68 81 98 75 105 66 77 103 53 90 77 73 101 83 109 53 83 97 57 88 100 50 116 10 116 57 55 121 69 50 119 76 47 87 78 112 79 47 50 122 98 87 56 67 107 75 111 51 76 89 88 77 80 51 75 71 73 47 87 103 99 55 114 43 90 115 113 67 109 66 121 79 87 105 54 49 67 48 50 43 73 113 89 73 83 111 112 110 10 83 55 103 86 122 98 50 107 78 79 114 106 53 111 99 67 87 121 53 80 86 70 86 108 100 116 67 55 72 54 73 79 102 50 98 48 99 103 102 90 102 109 90 73 88 69 76 111 65 109 75 83 68 116 67 72 111 43 50 113 117 74 43 80 10 53 54 78 72 52 115 113 85 43 88 98 108 110 120 97 106 69 66 86 87 121 73 117 66 120 76 108 100 49 55 116 48 52 51 110 56 73 71 109 107 68 56 54 73 107 43 71 82 57 74 80 89 74 111 67 88 89 82 54 103 51 82 76 80 10 104 54 104 48 79 53 105 49 86 99 116 118 54 66 67 47 111 68 99 117 112 110 101 118 117 89 99 97 79 103 55 72 107 114 99 82 56 51 100 73 120 52 100 116 116 119 69 81 72 70 53 52 43 99 116 102 70 117 52 53 105 65 70 108 10 77 102 87 108 111 117 74 102 109 83 54 79 66 86 89 52 78 103 116 116 101 111 71 78 113 88 86 67 87 57 69 77 122 119 84 113 110 107 121 52 57 66 76 51 110 52 112 106 85 105 106 89 115 53 105 107 77 120 89 61 10 45 45 45 45 45 69 78 68 32 67 69 82 84 73 70 73 67 65 84 69 45 45 45 45 45 10] <nil>}
RemoteClusterService 2015-04-16T20:29:51.826-07:00 [INFO] Deleting remote cluster with reference name=remote_cluster_C1-C2
RemoteClusterService 2015-04-16T20:29:51.826-07:00 [INFO] Adding remote cluster with referenceId remoteCluster/b58c5f4927e19f6bee1a426f151f4622
RemoteClusterService 2015-04-16T20:29:51.910-07:00 [INFO] Result from validate remote cluster call: err=<nil>, statusCode=200
MigrationService 2015-04-16T20:29:51.959-07:00 [INFO] Done with migrating remote cluster with name=remote_cluster_C1-C2. errorList=[]
but ns_server.reports.log later shows
=========================ERROR REPORT=========================
{mochiweb_socket_server,297,{acceptor_error,
{error,accept_failed}
}}
[error_logger:error,2015-04-16T20:42:59.206,ns_1@10.1.2.11:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]SSL: certify: tls_connection.erl:375:Fatal error: bad certificate
[error_logger:error,2015-04-16T20:42:59.206,ns_1@10.1.2.11:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================ERROR REPORT=========================
application: mochiweb
"Accept failed error"
"{error,{tls_alert,\"bad certificate\"}}"
[error_logger:error,2015-04-16T20:42:59.206,ns_1@10.1.2.11:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
Migration on C2-
MigrationService 2015-04-16T20:29:52.803-07:00 [INFO] Starting to migrate remote cluster
MigrationService 2015-04-16T20:29:52.803-07:00 [INFO] data=map[hostname:10.1.2.11:8091 username:Administrator uuid:7465eb07aa9978212edf0fa868240601 deleted:false password:password demandEncryption:true certificate:----BEGIN CERTIFICATE----
MIICmDCCAYKgAwIBAgIIE9WvNnWyXyswCwYJKoZIhvcNAQEFMAwxCjAIBgNVBAMT
ASowHhcNMTMwMTAxMDAwMDAwWhcNNDkxMjMxMjM1OTU5WjAMMQowCAYDVQQDEwEq
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvaAH9ijLP1ZJcyhqkPve
pXTKi95H+S3lJIx54PViyeI4m+7Rak9iuSHPCvdODN3BgRYrOWKu8VFU/58ElR9d
KrG7b1hg+LDPi5ia810DN0KKqPIal5WIpX99Ft+x1XXjQwabt2BcoCWlH+8hBCDG
anAdNG9CG8JNTHbcYIY//QieZV4VpnaWXUyBkDyRLIX/oTbpNQZBMrcjhPea+VsY
29JM5m4SAhWc7r+h5GkJtU7wzzWv1f3NdAxiA1SDGQOiv+Qc7/WYOe6zTOxdctRX
Tj51ZC8+/5XteVpmTTIul1kT+f3h1iMD2xOpj0jGit6qJ91LEK7qfBK00vqxZtbG
YQIDAQABowIwADALBgkqhkiG9w0BAQUDggEBAAHMv5HnHlKrheNl2xFr+HessR7Z
1evF5Td4fAVZDvfczGDiniqv5lbD0IwE+f9R6zKuAg+aDTVTbz2cLap2WztNwtZ+
vSDlMvMl5P8FnrdWtx84KZi9HI+xl2UE52k1YgJgTSaYFfe8SCFUnB5Ru+14fiUY
lpFdBEkTQmotUJWA7lEX8hHGEWIeLGberweBoby5WI0EaaUop7SCOcUxpZkGoF9e
THbRZkmwGh1r1C68e9vJgHSzZnrdXbQ3VyNEbNhlbamOBZgETiVwmVEiYBk0cxwQ
PpEpSEr0Sdvyt/pot4YSqnbzIIfGY0ylmgXl5Zu1G8LhwXdYeO2rc+ELA2g=
----END CERTIFICATE----
name:remote_cluster_C2-C1]
MigrationService 2015-04-16T20:29:52.803-07:00 [INFO] Remote cluster constructed = &
{remoteCluster/7465eb07aa9978212edf0fa868240601 7465eb07aa9978212edf0fa868240601 remote_cluster_C2-C1 10.1.2.11:8091 Administrator password true [45 45 45 45 45 66 69 71 73 78 32 67 69 82 84 73 70 73 67 65 84 69 45 45 45 45 45 10 77 73 73 67 109 68 67 67 65 89 75 103 65 119 73 66 65 103 73 73 69 57 87 118 78 110 87 121 88 121 115 119 67 119 89 74 75 111 90 73 104 118 99 78 65 81 69 70 77 65 119 120 67 106 65 73 66 103 78 86 66 65 77 84 10 65 83 111 119 72 104 99 78 77 84 77 119 77 84 65 120 77 68 65 119 77 68 65 119 87 104 99 78 78 68 107 120 77 106 77 120 77 106 77 49 79 84 85 53 87 106 65 77 77 81 111 119 67 65 89 68 86 81 81 68 69 119 69 113 10 77 73 73 66 73 106 65 78 66 103 107 113 104 107 105 71 57 119 48 66 65 81 69 70 65 65 79 67 65 81 56 65 77 73 73 66 67 103 75 67 65 81 69 65 118 97 65 72 57 105 106 76 80 49 90 74 99 121 104 113 107 80 118 101 10 112 88 84 75 105 57 53 72 43 83 51 108 74 73 120 53 52 80 86 105 121 101 73 52 109 43 55 82 97 107 57 105 117 83 72 80 67 118 100 79 68 78 51 66 103 82 89 114 79 87 75 117 56 86 70 85 47 53 56 69 108 82 57 100 10 75 114 71 55 98 49 104 103 43 76 68 80 105 53 105 97 56 49 48 68 78 48 75 75 113 80 73 97 108 53 87 73 112 88 57 57 70 116 43 120 49 88 88 106 81 119 97 98 116 50 66 99 111 67 87 108 72 43 56 104 66 67 68 71 10 97 110 65 100 78 71 57 67 71 56 74 78 84 72 98 99 89 73 89 47 47 81 105 101 90 86 52 86 112 110 97 87 88 85 121 66 107 68 121 82 76 73 88 47 111 84 98 112 78 81 90 66 77 114 99 106 104 80 101 97 43 86 115 89 10 50 57 74 77 53 109 52 83 65 104 87 99 55 114 43 104 53 71 107 74 116 85 55 119 122 122 87 118 49 102 51 78 100 65 120 105 65 49 83 68 71 81 79 105 118 43 81 99 55 47 87 89 79 101 54 122 84 79 120 100 99 116 82 88 10 84 106 53 49 90 67 56 43 47 53 88 116 101 86 112 109 84 84 73 117 108 49 107 84 43 102 51 104 49 105 77 68 50 120 79 112 106 48 106 71 105 116 54 113 74 57 49 76 69 75 55 113 102 66 75 48 48 118 113 120 90 116 98 71 10 89 81 73 68 65 81 65 66 111 119 73 119 65 68 65 76 66 103 107 113 104 107 105 71 57 119 48 66 65 81 85 68 103 103 69 66 65 65 72 77 118 53 72 110 72 108 75 114 104 101 78 108 50 120 70 114 43 72 101 115 115 82 55 90 10 49 101 118 70 53 84 100 52 102 65 86 90 68 118 102 99 122 71 68 105 110 105 113 118 53 108 98 68 48 73 119 69 43 102 57 82 54 122 75 117 65 103 43 97 68 84 86 84 98 122 50 99 76 97 112 50 87 122 116 78 119 116 90 43 10 118 83 68 108 77 118 77 108 53 80 56 70 110 114 100 87 116 120 56 52 75 90 105 57 72 73 43 120 108 50 85 69 53 50 107 49 89 103 74 103 84 83 97 89 70 102 101 56 83 67 70 85 110 66 53 82 117 43 49 52 102 105 85 89 10 108 112 70 100 66 69 107 84 81 109 111 116 85 74 87 65 55 108 69 88 56 104 72 71 69 87 73 101 76 71 98 101 114 119 101 66 111 98 121 53 87 73 48 69 97 97 85 111 112 55 83 67 79 99 85 120 112 90 107 71 111 70 57 101 10 84 72 98 82 90 107 109 119 71 104 49 114 49 67 54 56 101 57 118 74 103 72 83 122 90 110 114 100 88 98 81 51 86 121 78 69 98 78 104 108 98 97 109 79 66 90 103 69 84 105 86 119 109 86 69 105 89 66 107 48 99 120 119 81 10 80 112 69 112 83 69 114 48 83 100 118 121 116 47 112 111 116 52 89 83 113 110 98 122 73 73 102 71 89 48 121 108 109 103 88 108 53 90 117 49 71 56 76 104 119 88 100 89 101 79 50 114 99 43 69 76 65 50 103 61 10 45 45 45 45 45 69 78 68 32 67 69 82 84 73 70 73 67 65 84 69 45 45 45 45 45 10] <nil>}
RemoteClusterService 2015-04-16T20:29:52.803-07:00 [INFO] Deleting remote cluster with reference name=remote_cluster_C2-C1
RemoteClusterService 2015-04-16T20:29:52.803-07:00 [INFO] Adding remote cluster with referenceId remoteCluster/7465eb07aa9978212edf0fa868240601
RemoteClusterService 2015-04-16T20:29:52.907-07:00 [INFO] Result from validate remote cluster call: err=Get https://10.1.2.11:18091/pools: x509: cannot validate certificate for 10.1.2.11 because it doesn't contain any IP SANs, statusCode=0
Because migration on C2 failed, it fell back to erlang xdcr and started replicating data to C1. C1's migration passed but replication later failed because of "bad certificate" error. Why was migration at C1 successful?
Critical