Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25796

[XDCR] metadata migration completed with warnings during offline upgrade

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.0.0
    • 5.0.0
    • XDCR
    • Untriaged
    • No

    Description

      ./testrunner -i ~/testrunner/ini_files/xdcr-upgrade.ini -t "xdcr.upgradeXDCR.UpgradeTests.offline_cluster_upgrade,initial_version=3.1.5-1859,bucket_topology=default:1>2;sasl_bucket_1:1><2;standard_bucket_1:1<2,upgrade_nodes=dest;src,demand_encryption=1,upgrade_version=5.0.0-3497,skip_cleanup=True"

      Above test fails when doing an offline upgrade from 3.1.5-1859 to 5.0.0-3497 - metadata migration completed with warnings:

      2017-08-24T00:36:36.259Z INFO GOXDCR.MigrationSvc: Starting to migrate replication doc
      2017-08-24T00:36:36.259Z INFO GOXDCR.MigrationSvc: data=map[id:fce1c198403c0b1c6ad8ad86f4ecdc2f/sasl_bucket_1/sasl_bucket_1 type:xdc-xmem source:sasl_bucket_1 target:/remoteClusters/fce1c198403c0b1c6ad8ad86f4ecdc2f/buckets/sasl_bucket_1 continuous:true]
      2017-08-24T00:36:36.261Z ERRO GOXDCR.RemClusterSvc: Failed to connect to 10.111.170.103:8091, err=tls: oversized record received with length 20527
      2017-08-24T00:36:36.261Z ERRO GOXDCR.RemClusterSvc: Failed to get client for request, err=tls: oversized record received with length 20527, req=&{GET https://10.111.170.103:8091/pools/default HTTP/1.1 1 1 map[User-Agent:[] Authorization:[Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==] Content-Type:[application/x-www-form-urlencoded]] {} 0x6f6dd0 0 [] false 10.111.170.103:8091 map[] map[] <nil> map[]   <nil> <nil> <nil> <nil>}
      2017-08-24T00:36:36.262Z WARN GOXDCR.RemClusterSvc: When refreshing remote cluster reference remoteCluster/7_QKWkiAOhQY2hXq7hWc0xxap9lhhMMks_2VJZ6sZnE=, skipping node 10.111.170.103:8091 since it is not accessible. err=Failed on calling host=10.111.170.103:8091, path=/pools/default, err=tls: oversized record received with length 20527, statusCode=0
      2017-08-24T00:36:36.262Z ERRO GOXDCR.RemClusterSvc: Failed to refresh remote cluster reference remoteCluster/7_QKWkiAOhQY2hXq7hWc0xxap9lhhMMks_2VJZ6sZnE= since none of the nodes in target node list is accessible. node list = [10.111.170.103:8091]
       
      2017-08-24T00:36:36.273Z ERRO GOXDCR.MigrationSvc: Failed to connect to 10.111.170.103:8091, err=tls: oversized record received with length 20527
      2017-08-24T00:36:36.273Z ERRO GOXDCR.MigrationSvc: Failed to get client for request, err=tls: oversized record received with length 20527, req=&{GET https://10.111.170.103:8091/pools/default/buckets/sasl_bucket_1 HTTP/1.1 1 1 map[Content-Type:[application/x-www-form-urlencoded] User-Agent:[] Authorization:[Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==]] {} 0x6f6dd0 0 [] false 10.111.170.103:8091 map[] map[] <nil> map[]   <nil> <nil> <nil> <nil>}
      2017-08-24T00:36:36.273Z ERRO GOXDCR.MigrationSvc: Failed to get bucket info for bucket 'sasl_bucket_1'. host=10.111.170.103:8091, err=tls: oversized record received with length 20527, statusCode=0
      2017-08-24T00:36:36.273Z INFO GOXDCR.MigrationSvc: Done with converting replication settings to goxdcr settings. old settings=map[source:sasl_bucket_1 target:/remoteClusters/fce1c198403c0b1c6ad8ad86f4ecdc2f/buckets/sasl_bucket_1 continuous:true id:fce1c198403c0b1c6ad8ad86f4ecdc2f/sasl_bucket_1/sasl_bucket_1 type:xdc-xmem]
       new settings=map[replication_type:xmem target_nozzle_per_node:64]
       errorList=[]
      2017-08-24T00:36:36.273Z INFO GOXDCR.MigrationSvc: Replication spec constructed = &{fce1c198403c0b1c6ad8ad86f4ecdc2f/sasl_bucket_1/sasl_bucket_1 f12nbz5wlocc7saeGE9CFg== sasl_bucket_1 e7e3339e068913f32bda796c9e3cb398 fce1c198403c0b1c6ad8ad86f4ecdc2f sasl_bucket_1  0xc4201250e0 <nil>}
      2017-08-24T00:36:36.273Z INFO GOXDCR.ReplSpecSvc: Start AddReplicationSpec, spec=&{fce1c198403c0b1c6ad8ad86f4ecdc2f/sasl_bucket_1/sasl_bucket_1 f12nbz5wlocc7saeGE9CFg== sasl_bucket_1 e7e3339e068913f32bda796c9e3cb398 fce1c198403c0b1c6ad8ad86f4ecdc2f sasl_bucket_1  0xc4201250e0 <nil>}, additionalInfo=
      2017-08-24T00:36:36.273Z INFO GOXDCR.ReplSpecSvc: Adding it to metadata store...
      2017-08-24T00:36:36.277Z INFO GOXDCR.MigrationSvc: Done with migrating replication doc with id=fce1c198403c0b1c6ad8ad86f4ecdc2f/sasl_bucket_1/sasl_bucket_1. fatalErrorList=[], mildErrorList=[Error retrieving target bucket uuid for replication doc with sourceBucket=sasl_bucket_1, targetClusterUuid=fce1c198403c0b1c6ad8ad86f4ecdc2f, targetBucket=sasl_bucket_1. err=Failed to get bucket info.
      ]
      2017-08-24T00:36:36.277Z INFO GOXDCR.MigrationSvc: Starting to migrate replication doc
      2017-08-24T00:36:36.277Z INFO GOXDCR.MigrationSvc: data=map[id:fce1c198403c0b1c6ad8ad86f4ecdc2f/default/default type:xdc-xmem source:default target:/remoteClusters/fce1c198403c0b1c6ad8ad86f4ecdc2f/buckets/default continuous:true]
      2017-08-24T00:36:36.280Z ERRO GOXDCR.RemClusterSvc: Failed to connect to 10.111.170.103:8091, err=tls: oversized record received with length 20527
      2017-08-24T00:36:36.281Z ERRO GOXDCR.RemClusterSvc: Failed to get client for request, err=tls: oversized record received with length 20527, req=&{GET https://10.111.170.103:8091/pools/default HTTP/1.1 1 1 map[Content-Type:[application/x-www-form-urlencoded] User-Agent:[] Authorization:[Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==]] {} 0x6f6dd0 0 [] false 10.111.170.103:8091 map[] map[] <nil> map[]   <nil> <nil> <nil> <nil>}
      2017-08-24T00:36:36.281Z WARN GOXDCR.RemClusterSvc: When refreshing remote cluster reference remoteCluster/7_QKWkiAOhQY2hXq7hWc0xxap9lhhMMks_2VJZ6sZnE=, skipping node 10.111.170.103:8091 since it is not accessible. err=Failed on calling host=10.111.170.103:8091, path=/pools/default, err=tls: oversized record received with length 20527, statusCode=0
      2017-08-24T00:36:36.281Z ERRO GOXDCR.RemClusterSvc: Failed to refresh remote cluster reference remoteCluster/7_QKWkiAOhQY2hXq7hWc0xxap9lhhMMks_2VJZ6sZnE= since none of the nodes in target node list is accessible. node list = [10.111.170.103:8091]
       
      2017-08-24T00:36:36.291Z ERRO GOXDCR.MigrationSvc: Failed to connect to 10.111.170.103:8091, err=tls: oversized record received with length 20527
      2017-08-24T00:36:36.291Z ERRO GOXDCR.MigrationSvc: Failed to get client for request, err=tls: oversized record received with length 20527, req=&{GET https://10.111.170.103:8091/pools/default/buckets/default HTTP/1.1 1 1 map[Content-Type:[application/x-www-form-urlencoded] User-Agent:[] Authorization:[Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==]] {} 0x6f6dd0 0 [] false 10.111.170.103:8091 map[] map[] <nil> map[]   <nil> <nil> <nil> <nil>}
      2017-08-24T00:36:36.291Z ERRO GOXDCR.MigrationSvc: Failed to get bucket info for bucket 'default'. host=10.111.170.103:8091, err=tls: oversized record received with length 20527, statusCode=0
      2017-08-24T00:36:36.291Z INFO GOXDCR.MigrationSvc: Done with converting replication settings to goxdcr settings. old settings=map[source:default target:/remoteClusters/fce1c198403c0b1c6ad8ad86f4ecdc2f/buckets/default continuous:true id:fce1c198403c0b1c6ad8ad86f4ecdc2f/default/default type:xdc-xmem]
       new settings=map[replication_type:xmem target_nozzle_per_node:64]
       errorList=[]
      2017-08-24T00:36:36.291Z INFO GOXDCR.MigrationSvc: Replication spec constructed = &{fce1c198403c0b1c6ad8ad86f4ecdc2f/default/default L2bhCwCO4P8F77Yjbe0HsQ== default 49bae31194c369995ec287fe6a3bf714 fce1c198403c0b1c6ad8ad86f4ecdc2f default  0xc420125540 <nil>}
      2017-08-24T00:36:36.291Z INFO GOXDCR.ReplSpecSvc: Start AddReplicationSpec, spec=&{fce1c198403c0b1c6ad8ad86f4ecdc2f/default/default L2bhCwCO4P8F77Yjbe0HsQ== default 49bae31194c369995ec287fe6a3bf714 fce1c198403c0b1c6ad8ad86f4ecdc2f default  0xc420125540 <nil>}, additionalInfo=
      2017-08-24T00:36:36.291Z INFO GOXDCR.ReplSpecSvc: Adding it to metadata store...
      2017-08-24T00:36:36.294Z INFO GOXDCR.MigrationSvc: Done with migrating replication doc with id=fce1c198403c0b1c6ad8ad86f4ecdc2f/default/default. fatalErrorList=[], mildErrorList=[Error retrieving target bucket uuid for replication doc with sourceBucket=default, targetClusterUuid=fce1c198403c0b1c6ad8ad86f4ecdc2f, targetBucket=default. err=Failed to get bucket info.
      ]
      2017-08-24T00:36:36.294Z INFO GOXDCR.MigrationSvc: Metadata migration completed with warnings. Some migrated metadata may be invalid. warnings = [Error retrieving target bucket uuid for replication doc with sourceBucket=sasl_bucket_1, targetClusterUuid=fce1c198403c0b1c6ad8ad86f4ecdc2f, targetBucket=sasl_bucket_1. err=Failed to get bucket info.
       Error retrieving target bucket uuid for replication doc with sourceBucket=default, targetClusterUuid=fce1c198403c0b1c6ad8ad86f4ecdc2f, targetBucket=default. err=Failed to get bucket info.
      ]
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty