Uploaded image for project: 'Java Couchbase JVM Core'
  1. Java Couchbase JVM Core
  2. JVMCBC-244

Incorrect data from get

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Major
    • .backlog
    • 1.1.4
    • None
    • None
    • -java 6
      -Couchbase Server 3.0.1
      -Couchbase java client api 2.1.4
      -ubuntu server
      -25 node cluster

    Description

      We have been running into an issue related to incorrect data while trying to ramp up in production.
      In the code, I'm using the non-blocking pattern (rxJava + Twitter Future).

      • Fetch a set of data for a set of keys
        -subscribe to the results and once completed set the twitter promise.
        -Compare the request keys and the result keys and the keys are different. The result appears to be from a different request.

      Current traffic is ~180k writes per a min and ~180k read per a min and we find at most one request would hit the scenario above and so it's very rare. We haven't been able to repro this in a test environment (probably since we are not hitting the same QPS or the data is different).

      Code to fetch data and subscribe:
      private def batchGet(ids: Iterable[String], keySpace: String): Future[Seq[JsonDocument]] = {
      val promise = new Promise[Seq[JsonDocument]]()
      val errorLogs =

      { ids.toString() }

      Observable.from(ids.toArray)
      .flatMap(new Func1[String, Observable[JsonDocument]]() {
      override def call(id: String): Observable[JsonDocument] =

      { bucket.async().get(id) }

      }).toList()
      .single()
      .subscribe(new Action1[java.util.List[JsonDocument]] {
      override def call(t1: java.util.List[JsonDocument]): Unit =

      { promise.setValue(t1) }

      }, errorAction(promise, Seq.empty, keySpace, BATCHGET_STAT_TYPE, errorLogs), completeAction(promise, errorLogs))
      promise
      }

      Calling method:

      batchGet(keyMapping.keys, keySpace).map { seq =>
      seq.map { jsonDoc =>
      (keyMapping.get(jsonDoc.id()).getOrElse

      { val msg = "missing key: " + jsonDoc.id() + ". available keys: " + keyMapping.keySet logger.warning(msg) //We're find examples where the keys doesn't match the request in the log throw new CacheResultMisMatchException(msg) }

      ->
      (parseDoc(jsonDoc), jsonDoc))
      }.toMap
      }

      All our data types are immutable other than the twitter promise. The keyMapping.keys are used to fetch the items from Couchbase but we found results from Couchbase where the id wasn't in the set of keys that was requested. In the logs, it clearly shows it is unrelated (and not a mismatch in casing or spacing).

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            daschl Michael Nitschinger
            chan00 chan00
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty