Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47382

[CX] analytics fails to ingest on KV "rollback" by way of file restore

    XMLWordPrintable

Details

    Description

      Should the KV nodes in a couchbase cluster (local or remote) be restored to an earlier state, analytics will get into a reconnect loop, continuously trying to resume from a point in the future. Analytics needs to consider the current seqnos for each vbucket master when determining a rollback / resume point.

      e.g.

      2021-07-12T19:37:58.649+00:00 INFO CBAS.conductor.Fixer [FixerThread:cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23] Handling {"open-stream-response":"0x23: Rollback required", "rollback-seq":486680}
      2021-07-12T19:37:58.649+00:00 INFO CBAS.conductor.Conductor [FixerThread:cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23] Conductor.disconnect called.
      2021-07-12T19:37:58.649+00:00 DEBU CBAS.state.SessionState [FixerThread:cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23] SessionState{numPartitions=1024, uuid='ba2f298ff5c308e6ff5f4edf55db401a', streams=["1:0xd", "2:0xf", "3:0x12", "4:0xc", "5:0xe", "6:0x11", "7:0xb", "8:0x8", "9:0x9", "10:0xa", "11:0x10"]} (0x227c76b1): disconnected
      2021-07-12T19:37:58.649+00:00 INFO CBAS.conductor.Fixer [FixerThread:cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23] FixerThread:cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 has been poisoned
      2021-07-12T19:37:58.651+00:00 DEBU CBAS.adapter.CouchbaseConnector [Executor-6280:91a6d2e34b3a720e267219776c7d3073] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 stopping...
      2021-07-12T19:38:03.648+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 stopping...
      2021-07-12T19:38:03.648+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 disconnecting...
      2021-07-12T19:38:03.648+00:00 INFO CBAS.dcp.Client [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] Disconnecting the client: cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 started
      2021-07-12T19:38:03.648+00:00 INFO CBAS.dcp.Client [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] Shutting down the environment of the client: cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23
      2021-07-12T19:38:03.648+00:00 INFO CBAS.dcp.Client [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] Disconnecting the client: cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 completed
      2021-07-12T19:38:03.648+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 ...disconnected
      2021-07-12T19:38:03.648+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 failing any pending seqno requests
      2021-07-12T19:38:03.648+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 ...stopped
      2021-07-12T19:38:03.649+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 closing...
      2021-07-12T19:38:03.649+00:00 DEBU CBAS.adapter.CouchbaseConnector [Executor-6237:91a6d2e34b3a720e267219776c7d3073] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 stopping...
      2021-07-12T19:38:03.649+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 closed...
      2021-07-12T19:38:03.775+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.447:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 calling client.connect()
      2021-07-12T19:38:03.795+00:00 DEBU CBAS.netty.DcpConnectHandler [nioEventLoopGroup-3-6] DCP Connection opened with Name "cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23" against Node /172.31.24.177:11210
      2021-07-12T19:38:03.802+00:00 DEBU CBAS.netty.DcpConnectHandler [nioEventLoopGroup-3-1] DCP Connection opened with Name "cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23" against Node /172.31.24.253:11210
      2021-07-12T19:38:03.810+00:00 DEBU CBAS.netty.DcpConnectHandler [nioEventLoopGroup-3-8] DCP Connection opened with Name "cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23" against Node /172.31.28.201:11210
      2021-07-12T19:38:03.817+00:00 DEBU CBAS.netty.DcpConnectHandler [nioEventLoopGroup-3-13] DCP Connection opened with Name "cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23" against Node /172.31.28.48:11210
      2021-07-12T19:38:03.826+00:00 DEBU CBAS.adapter.CouchbaseConnector [Executor-6279:91a6d2e34b3a720e267219776c7d3073] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 stopping...
      2021-07-12T19:38:03.830+00:00 INFO CBAS.conductor.Fixer [FixerThread:cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23] Handling {"open-stream-response":"0x23: Rollback required", "rollback-seq":484972}
      ...
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.1.0-1082 contains analytics-dcp-client commit 183100f with commit message:
            MB-47382: improve stream end logging, enable bucket-level seqno fetches

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1082 contains analytics-dcp-client commit 183100f with commit message: MB-47382 : improve stream end logging, enable bucket-level seqno fetches

            Build couchbase-server-7.1.0-1082 contains cbas-core commit 3d7fcfe with commit message:
            MB-47382: add test

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1082 contains cbas-core commit 3d7fcfe with commit message: MB-47382 : add test

            Build couchbase-server-7.1.0-1082 contains cbas-core commit 0b5edc2 with commit message:
            MB-47382: validate component at latest bucket uuid seqno <= current max seqno in master

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1082 contains cbas-core commit 0b5edc2 with commit message: MB-47382 : validate component at latest bucket uuid seqno <= current max seqno in master
            ritam.sharma Ritam Sharma added a comment -

            We need to take this for 7.0.2

            cc - Wayne Siu Till Westmann

            ritam.sharma Ritam Sharma added a comment - We need to take this for 7.0.2 cc - Wayne Siu Till Westmann
            umang.agrawal Umang added a comment -

            Verified with Enterprise Edition 7.0.2 build 6593

            umang.agrawal Umang added a comment - Verified with Enterprise Edition 7.0.2 build 6593

            People

              umang.agrawal Umang
              michael.blow Michael Blow
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty