Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47382

[CX] analytics fails to ingest on KV "rollback" by way of file restore

    XMLWordPrintable

Details

    Description

      Should the KV nodes in a couchbase cluster (local or remote) be restored to an earlier state, analytics will get into a reconnect loop, continuously trying to resume from a point in the future. Analytics needs to consider the current seqnos for each vbucket master when determining a rollback / resume point.

      e.g.

      2021-07-12T19:37:58.649+00:00 INFO CBAS.conductor.Fixer [FixerThread:cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23] Handling {"open-stream-response":"0x23: Rollback required", "rollback-seq":486680}
      		 
      2021-07-12T19:37:58.649+00:00 INFO CBAS.conductor.Conductor [FixerThread:cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23] Conductor.disconnect called.
      		 
      2021-07-12T19:37:58.649+00:00 DEBU CBAS.state.SessionState [FixerThread:cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23] SessionState{numPartitions=1024, uuid='ba2f298ff5c308e6ff5f4edf55db401a', streams=["1:0xd", "2:0xf", "3:0x12", "4:0xc", "5:0xe", "6:0x11", "7:0xb", "8:0x8", "9:0x9", "10:0xa", "11:0x10"]} (0x227c76b1): disconnected
      		 
      2021-07-12T19:37:58.649+00:00 INFO CBAS.conductor.Fixer [FixerThread:cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23] FixerThread:cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 has been poisoned
      		 
      2021-07-12T19:37:58.651+00:00 DEBU CBAS.adapter.CouchbaseConnector [Executor-6280:91a6d2e34b3a720e267219776c7d3073] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 stopping...
      		 
      2021-07-12T19:38:03.648+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 stopping...
      		 
      2021-07-12T19:38:03.648+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 disconnecting...
      		 
      2021-07-12T19:38:03.648+00:00 INFO CBAS.dcp.Client [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] Disconnecting the client: cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 started
      		 
      2021-07-12T19:38:03.648+00:00 INFO CBAS.dcp.Client [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] Shutting down the environment of the client: cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23
      		 
      2021-07-12T19:38:03.648+00:00 INFO CBAS.dcp.Client [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] Disconnecting the client: cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 completed
      		 
      2021-07-12T19:38:03.648+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 ...disconnected
      		 
      2021-07-12T19:38:03.648+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 failing any pending seqno requests
      		 
      2021-07-12T19:38:03.648+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 ...stopped
      		 
      2021-07-12T19:38:03.649+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 closing...
      		 
      2021-07-12T19:38:03.649+00:00 DEBU CBAS.adapter.CouchbaseConnector [Executor-6237:91a6d2e34b3a720e267219776c7d3073] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 stopping...
      		 
      2021-07-12T19:38:03.649+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.446:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 closed...
      		 
      2021-07-12T19:38:03.775+00:00 DEBU CBAS.adapter.CouchbaseConnector [SAO:JID:2.447:TAID:TID:ANID:ODID:24:0:23:0:(Local/bench)[23]:BO] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 calling client.connect()
      		 
      2021-07-12T19:38:03.795+00:00 DEBU CBAS.netty.DcpConnectHandler [nioEventLoopGroup-3-6] DCP Connection opened with Name "cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23" against Node /172.31.24.177:11210
      		 
      2021-07-12T19:38:03.802+00:00 DEBU CBAS.netty.DcpConnectHandler [nioEventLoopGroup-3-1] DCP Connection opened with Name "cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23" against Node /172.31.24.253:11210
      		 
      2021-07-12T19:38:03.810+00:00 DEBU CBAS.netty.DcpConnectHandler [nioEventLoopGroup-3-8] DCP Connection opened with Name "cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23" against Node /172.31.28.201:11210
      		 
      2021-07-12T19:38:03.817+00:00 DEBU CBAS.netty.DcpConnectHandler [nioEventLoopGroup-3-13] DCP Connection opened with Name "cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23" against Node /172.31.28.48:11210
      		 
      2021-07-12T19:38:03.826+00:00 DEBU CBAS.adapter.CouchbaseConnector [Executor-6279:91a6d2e34b3a720e267219776c7d3073] cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23 stopping...
      		 
      2021-07-12T19:38:03.830+00:00 INFO CBAS.conductor.Fixer [FixerThread:cbas:Local:bench:91a6d2e34b3a720e267219776c7d3073:23] Handling {"open-stream-response":"0x23: Rollback required", "rollback-seq":484972}
      		 
      ...

      Attachments

        Issue Links

          is cloned by

          Bug - A problem which impairs or prevents the functions of the product. MB-47403 [CX] analytics fails to ingest on KV "rollback" by way of file restore

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              umang.agrawal Umang
              michael.blow Michael Blow
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty