XDCR - panic in responseLookup ref counter

============================================================================

Running collections migration backfill test case with brokenmapping

============================================================================

C1 has already been set up - skipping setup

C2 has already been set up - skipping setup

For cluster C1 CREATING bucket B0 with properties ([ramQuotaMB]="100" )

For cluster C1 CREATING bucket B1 with properties ([CompressionMode]="Active" [ramQuotaMB]="100" )

For cluster C1 bucket B1 CREATING scope S1

For cluster C1 bucket B1 scope S1 CREATING collection col1

For cluster C1 bucket B1 scope S1 CREATING collection col2

For cluster C1 bucket B1 CREATING scope S2

For cluster C1 bucket B1 scope S2 CREATING collection col1

For cluster C1 bucket B1 scope S2 CREATING collection col2

For cluster C1 bucket B1 scope S2 CREATING collection col3

For cluster C2 CREATING bucket B2 with properties ([ramQuotaMB]="100" )

For cluster C2 bucket B2 CREATING scope S3

For cluster C2 bucket B2 scope S3 CREATING collection col1

For cluster C2 bucket B2 scope S3 CREATING collection col3

Temporarily updating targetManifestPull from 5 to 5

Sleeping 10 seconds for XDCR to reboot before checking...

Loading beer sample on source cluster...

Executing cbimport on bucket B1 for file /Users/neil.huang/source/couchbase/install/samples/beer-sample.zip...

JSON `/Users/neil.huang/source/couchbase/install/samples/beer-sample.zip` imported to `localhost:9000` successfully

Documents imported: 7303 Documents failed: 0

Creating remote cluster reference from C1 to C2

{"connectivityErrors":null,"deleted":false,"hostname":"127.0.0.1:9001","name":"C2","network_type":"","secureType":"none","uri":"/pools/default/remoteClusters/C2","username":"Administrator","uuid":"d2a23b1afe53b87ac8cc9253d26f7394","validateURI":"/pools/default/remoteClusters/C2?just_validate=1"}

Creating remote cluster reference from C2 to C1

{"connectivityErrors":null,"deleted":false,"hostname":"127.0.0.1:9000","name":"C1","network_type":"","secureType":"none","uri":"/pools/default/remoteClusters/C1","username":"Administrator","uuid":"777286543583acd8f78835b8a566ae08","validateURI":"/pools/default/remoteClusters/C1?just_validate=1"}

ReplicationID: "d2a23b1afe53b87ac8cc9253d26f7394/B1/B2" Rest-FriendlyID: "d2a23b1afe53b87ac8cc9253d26f7394%2FB1%2FB2"

Cluster C1 Bucket B0 collections manifest version: "1"

Manifest version 1

ScopeName "_default" (UID 0)

ScopeName "_default" CollectionName "_default" (UID 0)

ScopeName "_system" (UID 8)

ScopeName "_system" CollectionName "_query" (UID 9)

ScopeName "_system" CollectionName "_mobile" (UID 8)

Cluster C1 Bucket B1 collections manifest version: "8"

Manifest version 8

ScopeName "S2" (UID 10)

ScopeName "S2" CollectionName "col3" (UID 14)

ScopeName "S2" CollectionName "col2" (UID 13)

ScopeName "S2" CollectionName "col1" (UID 12)

ScopeName "S1" (UID 9)

ScopeName "S1" CollectionName "col2" (UID 11)

ScopeName "S1" CollectionName "col1" (UID 10)

ScopeName "_default" (UID 0)

ScopeName "_default" CollectionName "_default" (UID 0)

ScopeName "_system" (UID 8)

ScopeName "_system" CollectionName "_query" (UID 9)

ScopeName "_system" CollectionName "_mobile" (UID 8)

Cluster C2 Bucket B2 collections manifest version: "4"

Manifest version 4

ScopeName "S3" (UID 9)

ScopeName "S3" CollectionName "col3" (UID 11)

ScopeName "S3" CollectionName "col1" (UID 10)

ScopeName "_default" (UID 0)

ScopeName "_default" CollectionName "_default" (UID 0)

ScopeName "_system" (UID 8)

ScopeName "_system" CollectionName "_query" (UID 9)

ScopeName "_system" CollectionName "_mobile" (UID 8)

Waiting 15 seconds before checking changes left is 0

Changes left 14115 is not 0 yet...

Sleeping 10 seconds and retrying...

Changes left 10810 is not 0 yet...

Sleeping 10 seconds and retrying...

Changes left 7106 is not 0 yet...

Sleeping 10 seconds and retrying...

Changes left 3134 is not 0 yet...

Sleeping 10 seconds and retrying...

Checking item count 1 / 8

Item count for cluster C2 bucket B2: 920

Warning - requested count for Found following destination collection(s) missing (and will not get replicated to) is 23 or < 24, found 24

C1 has already been set up - skipping setup

C2 has already been set up - skipping setup

For cluster C1 CREATING bucket B0 with properties ([ramQuotaMB]="100" )

For cluster C1 CREATING bucket B1 with properties ([CompressionMode]="Active" [ramQuotaMB]="100" )

For cluster C2 CREATING bucket B2 with properties ([ramQuotaMB]="100" )

For cluster C2 bucket B2 scope S3 CREATING collection col2

Sleeping 30 seconds for discovery and automatic backfill

Cluster C1 Bucket B0 collections manifest version: "1"

Manifest version 1

ScopeName "_default" (UID 0)

ScopeName "_default" CollectionName "_default" (UID 0)

ScopeName "_system" (UID 8)

ScopeName "_system" CollectionName "_query" (UID 9)

ScopeName "_system" CollectionName "_mobile" (UID 8)

Cluster C1 Bucket B1 collections manifest version: "8"

Manifest version 8

ScopeName "S2" (UID 10)

ScopeName "S2" CollectionName "col3" (UID 14)

ScopeName "S2" CollectionName "col2" (UID 13)

ScopeName "S2" CollectionName "col1" (UID 12)

ScopeName "S1" (UID 9)

ScopeName "S1" CollectionName "col2" (UID 11)

ScopeName "S1" CollectionName "col1" (UID 10)

ScopeName "_default" (UID 0)

ScopeName "_default" CollectionName "_default" (UID 0)

ScopeName "_system" (UID 8)

ScopeName "_system" CollectionName "_query" (UID 9)

ScopeName "_system" CollectionName "_mobile" (UID 8)

Cluster C2 Bucket B2 collections manifest version: "5"

Manifest version 5

ScopeName "S3" (UID 9)

ScopeName "S3" CollectionName "col2" (UID 12)

ScopeName "S3" CollectionName "col3" (UID 11)

ScopeName "S3" CollectionName "col1" (UID 10)

ScopeName "_default" (UID 0)

ScopeName "_default" CollectionName "_default" (UID 0)

ScopeName "_system" (UID 8)

ScopeName "_system" CollectionName "_query" (UID 9)

ScopeName "_system" CollectionName "_mobile" (UID 8)

Checking item count 1 / 8

ERROR: Cluster C2 bucket B2 only has 1107 items. Expect 1412. Items are increasing...

Sleeping 10 seconds and retrying...

Checking item count 1 / 8

ERROR: Cluster C2 bucket B2 only has 1298 items. Expect 1412. Items are increasing...

Sleeping 10 seconds and retrying...

Checking item count 1 / 8

Item count for cluster C2 bucket B2: 1412

Found exactly 2 of "These collections need to backfill"

WARNING Found panics in /Users/neil.huang/source/couchbase/ns_server/logs/n_0

Cleaning up internal settings

// stores response and increases the reference count by 1

func (lookup *responseLookup) registerLookup(uniqueKey string, resp *base.SubdocLookupResponse) {

	if lookup == nil {

		return

	}

	lookup.responses[uniqueKey] = resp

	lookup.refCnter[resp.Resp]++

}