Data race between cancellation and retries

WARNING: DATA RACE
Read at 0x00c04a3e6ae0 by goroutine 1353:
  github.com/couchbase/gocbcore/v10.(*tracerComponent).StartCmdTrace()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/tracing.go:132 +0x64
  github.com/couchbase/gocbcore/v10.(*kvMux).requeueDirect()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/kvmux.go:375 +0x73
  github.com/couchbase/gocbcore/v10.(*kvMux).RequeueDirect()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/kvmux.go:371 +0x84
  github.com/couchbase/gocbcore/v10.(*kvMux).waitAndRetryOperation.func1()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/kvmux.go:730 +0x6a

Previous write at 0x00c04a3e6ae0 by goroutine 1335:
  github.com/couchbase/gocbcore/v10.stopCmdTrace()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/tracing.go:209 +0x136
  github.com/couchbase/gocbcore/v10.cancelReqTrace()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/tracing.go:218 +0xa6
  github.com/couchbase/gocbcore/v10.(*memdQRequest).internalCancel()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/memdqpackets.go:281 +0x1da
  github.com/couchbase/gocbcore/v10.(*memdQRequest).cancelWithCallback()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/memdqpackets.go:290 +0x44
  github.com/couchbase/gocbcore/v10.(*memdQRequest).Cancel()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/memdqpackets.go:309 +0x6d
  github.com/couchbase/gocb/v2.(*kvOpManagerCore).Wait()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/src/github.com/couchbase/gocb/kvopmanager_core.go:305 +0x201
  github.com/couchbase/gocb/v2.(*kvProviderCore).getOneReplica()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/src/github.com/couchbase/gocb/kvprovider_core.go:982 +0xfba
  github.com/couchbase/gocb/v2.(*kvProviderCore).GetAllReplicas.func1()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/src/github.com/couchbase/gocb/kvprovider_core.go:841 +0x204
  github.com/couchbase/gocb/v2.(*kvProviderCore).GetAllReplicas.func3()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/src/github.com/couchbase/gocb/kvprovider_core.go:849 +0x47

Goroutine 1353 (running) created at:
  github.com/couchbase/gocbcore/v10.(*kvMux).waitAndRetryOperation()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/kvmux.go:728 +0x15a
  github.com/couchbase/gocbcore/v10.(*kvMux).handleOpRoutingResp()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/kvmux.go:646 +0x610
  github.com/couchbase/gocbcore/v10.(*kvMux).handleOpRoutingResp-fm()
      <autogenerated>:1 +0x73
  github.com/couchbase/gocbcore/v10.(*memdClient).run.func2.1()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/memdclient.go:530 +0x117
  github.com/couchbase/gocbcore/v10.(*memdOpMap).Drain()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/memdopmap.go:88 +0x1281
  github.com/couchbase/gocbcore/v10.(*memdClient).run.func2()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/pkg/mod/github.com/couchbase/gocbcore/v10@v10.5.0/memdclient.go:525 +0x11aa

Goroutine 1335 (finished) created at:
  github.com/couchbase/gocb/v2.(*kvProviderCore).GetAllReplicas()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/src/github.com/couchbase/gocb/kvprovider_core.go:837 +0xa5a
  github.com/couchbase/gocb/v2.(*kvProviderCore).GetAnyReplica()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/src/github.com/couchbase/gocb/kvprovider_core.go:890 +0x573
  github.com/couchbase/gocb/v2.(*Collection).GetAnyReplica.func1()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/src/github.com/couchbase/gocb/collection_crud.go:359 +0x131
  github.com/couchbase/gocb/v2.autoOpControl[...]()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/src/github.com/couchbase/gocb/client.go:59 +0x12a
  github.com/couchbase/gocb/v2.(*Collection).GetAnyReplica()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/src/github.com/couchbase/gocb/collection_crud.go:354 +0x10f
  github.com/couchbase/gocb/v2.BenchmarkGetFromReplica.func1()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/src/github.com/couchbase/gocb/collection_crud_bench_test.go:140 +0x6d
  testing.(*B).RunParallel.func1()
      /home/couchbase/jenkins/workspace/golang/gocb/gocb-benchmarks/deps/go1.19.9/src/testing/benchmark.go:788 +0x1b6

I think this could happen for any op. We should only be accessing the cmd span in the processing lock, but we aren't.