Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Major
Fix Version/s: 5.0.0
Affects Version/s: 5.0.0
Component/s: XDCR
Labels:
- beta2-candidate

Triage:
Untriaged
Is this a Regression?:
No

Description

These are the data races that are found from running the XDCR_acceptance test.
The code changes should correspond to a custom gerrit build of:
git fetch http://review.couchbase.org/goxdcr refs/changes/68/77568/25 && git checkout FETCH_HEAD

==================

WARNING: DATA RACE

Read at 0x00c4200840b0 by goroutine 113:

  github.com/couchbase/goxdcr/pipeline.(*GenericPipeline).State()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline/generic_pipeline.go:579 +0x44

  github.com/couchbase/goxdcr/pipeline_svc.(*StatisticsManager).updateStatsOnce()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/statistics_manager.go:322 +0x76

  github.com/couchbase/goxdcr/pipeline_svc.(*StatisticsManager).updateStats()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/statistics_manager.go:288 +0x76b

Previous write at 0x00c4200840b0 by goroutine 65:

  github.com/couchbase/goxdcr/pipeline.(*GenericPipeline).SetState()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline/generic_pipeline.go:642 +0x11a

  github.com/couchbase/goxdcr/pipeline.(*GenericPipeline).Stop()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline/generic_pipeline.go:357 +0x1dd

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).StopPipelineInner()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:432 +0x1cb

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).update()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:781 +0x330

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).run()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:731 +0x340

Goroutine 113 (running) created at:

  github.com/couchbase/goxdcr/pipeline_svc.(*StatisticsManager).Start()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/statistics_manager.go:716 +0x4f2

  github.com/couchbase/goxdcr/pipeline_ctx.(*PipelineRuntimeCtx).Start()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_ctx/pipeline_runtimeCtx.go:70 +0x1f3

  github.com/couchbase/goxdcr/pipeline.(*GenericPipeline).Start()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline/generic_pipeline.go:207 +0xaff

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).StartPipeline()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:348 +0x4f6

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).update()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:804 +0xea2

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).run()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:714 +0xdf

Goroutine 65 (running) created at:

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).launchUpdater()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:518 +0x250

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).GetOrCreateReplicationStatus()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:552 +0x435

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).Update()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:568 +0xf8

  github.com/couchbase/goxdcr/replication_manager.(*ReplicationSpecChangeListener).launchPipelineUpdate()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/replication_manager/metakv_change_listener.go:215 +0x74

==================

==================

WARNING: DATA RACE

Write at 0x00c42032a310 by goroutine 143:

2017-05-25T10:21:57.210-07:00 INFO GOXDCR.PipelineMgr: Replication Status=name={4e10e68c87d10130ea104a6d9967b901/default/default}, errors={[]}, progress={Pipeline has been stopped}

  github.com/couchbase/goxdcr/pipeline.(*GenericPipeline).Stop.func1()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline/generic_pipeline.go:389 +0x92

Previous read at 0x00c42032a310 by goroutine 65:

  github.com/couchbase/goxdcr/pipeline.(*GenericPipeline).Stop()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline/generic_pipeline.go:401 +0xb63

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).StopPipelineInner()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:432 +0x1cb

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).update()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:781 +0x330

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).run()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:731 +0x340

Goroutine 143 (running) created at:

  github.com/couchbase/goxdcr/pipeline.(*GenericPipeline).Stop()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline/generic_pipeline.go:394 +0x9a5

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).StopPipelineInner()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:432 +0x1cb

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).update()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:781 +0x330

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).run()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:731 +0x340

Goroutine 65 (running) created at:

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).launchUpdater()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:518 +0x250

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).GetOrCreateReplicationStatus()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:552 +0x435

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).Update()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:568 +0xf8

  github.com/couchbase/goxdcr/replication_manager.(*ReplicationSpecChangeListener).launchPipelineUpdate()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/replication_manager/metakv_change_listener.go:215 +0x74

==================

==================

WARNING: DATA RACE

Read at 0x00c420084180 by goroutine 166:

  github.com/couchbase/goxdcr/pipeline.(*GenericPipeline).State()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline/generic_pipeline.go:579 +0x44

  github.com/couchbase/goxdcr/pipeline_svc.(*StatisticsManager).updateStatsOnce()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/statistics_manager.go:322 +0x76

  github.com/couchbase/goxdcr/pipeline_svc.(*StatisticsManager).updateStats()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/statistics_manager.go:288 +0x76b

Previous write at 0x00c420084180 by goroutine 65:

  github.com/couchbase/goxdcr/pipeline.(*GenericPipeline).SetState()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline/generic_pipeline.go:642 +0x11a

  github.com/couchbase/goxdcr/pipeline.(*GenericPipeline).Start()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline/generic_pipeline.go:272 +0x1507

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).StartPipeline()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:348 +0x4f6

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).update()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:804 +0xea2

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).run()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:731 +0x340

Goroutine 166 (running) created at:

  github.com/couchbase/goxdcr/pipeline_svc.(*StatisticsManager).Start()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/statistics_manager.go:716 +0x4f2

  github.com/couchbase/goxdcr/pipeline_ctx.(*PipelineRuntimeCtx).Start()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_ctx/pipeline_runtimeCtx.go:70 +0x1f3

  github.com/couchbase/goxdcr/pipeline.(*GenericPipeline).Start()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline/generic_pipeline.go:207 +0xaff

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).StartPipeline()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:348 +0x4f6

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).update()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:804 +0xea2

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).run()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:731 +0x340

Goroutine 65 (running) created at:

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).launchUpdater()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:518 +0x250

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).GetOrCreateReplicationStatus()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:552 +0x435

  github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).Update()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:568 +0xf8

  github.com/couchbase/goxdcr/replication_manager.(*ReplicationSpecChangeListener).launchPipelineUpdate()

      /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/replication_manager/metakv_change_listener.go:215 +0x74

==================

Attachments

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: Arunkumar Senthilnathan (Inactive)

Reporter:: Neil Huang

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 25/May/17 11:59 AM

Updated:: 12/Jun/17 2:54 AM

Resolved:: 30/May/17 5:37 PM

Gerrit Reviews

There are no open Gerrit changes

Show There is 1 closed Gerrit change

Hide There is 1 closed Gerrit change

MB-23929 Changes to Pipeline Manager's update() method to minimize possible corner cases - Fix up unit test and also added singular updater creation. - Refactor update() method. - Use timer for scheduling future updates instead of depending on states. - Fixed mock test framework dealing with replicationStatus getting and setting. - Added unit test to test refactored PipelineMgr's update method and to test the scheduler. - Updated mock for pipeline manager and pipeline updater. - Rename state_lock to pipelineUpdaterLock - Use upgradable RW lock instead of atomic variable - Cleaned up mocking code to have generic setup functions - Implemented parallel test case, and double error test cases, and error injection test cases. - Added unit test to test the complete update path with pipeline stop, quiesce, and restart - Fixed a few race conditions after running golang race detector (MB-24555): Gerrit Review:

Pipeline Manager potential race conditions

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty