Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-52470

[XDCR] panic after capi replication is created

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • Yes

    Description

      Trying to create a capi replication from 2 node 6.6.5 source cluster to 2 node 6.6.5 target cluster (one of the target nodes is configured with couchbase ES connector). Seeing this panic in goxdcr.log:

      2022-06-07T14:01:10.848-07:00 INFO GOXDCR.XDCRRouter: Router_dcp_3f98ba3c50855281ed0b87c551a0c3c9/default/default_172.23.123.135:11210_0 created with 1 downstream parts isHighReplication=true and filter=<nil>
      2022-06-07T14:01:10.848-07:00 INFO GOXDCR.XDCRFactory: Constructed router Router_dcp_3f98ba3c50855281ed0b87c551a0c3c9/default/default_172.23.123.135:11210_0
      panic: interface conversion: *parts.CapiNozzle is not common.OutNozzle: missing method SetUpstreamObjRecycler
       
      goroutine 82 [running]:
      github.com/couchbase/goxdcr/factory.(*XDCRFactory).NewPipeline(0xc0002260c0, 0xc00019a2d0, 0x30, 0xc0001a73e0, 0x0, 0x0, 0x88, 0x88)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/factory/xdcr_factory.go:215 +0x13cc
      github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).StartPipeline(0xc0001c6b40, 0xc00019a2d0, 0x30, 0xc00035e090)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:397 +0x332
      github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).update(0xc00011c0f0, 0xaff358)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:1095 +0x9e7
      github.com/couchbase/goxdcr/pipeline_manager.(*PipelineUpdater).run(0xc00011c0f0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:951 +0x41a
      created by github.com/couchbase/goxdcr/pipeline_manager.(*PipelineManager).launchUpdater
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_manager/pipeline_manager.go:626 +0x176
      

      Seeing this crash on target's reports.log:

      [error_logger:error,2022-06-07T13:57:13.078-07:00,ns_1@172.23.107.135:error_logger<0.32.0>:ale_error_logger_handler:do_log:203]** Generic server <0.28207.3> terminating
      ** Last message in was {tcp_closed,#Port<0.43481>}
      ** When Server state == {state,#Port<0.43481>,
                                  {producer,
                                      "replication:ns_1@172.23.123.134->ns_1@172.23.107.135:default",
                                      'ns_1@172.23.123.134',"default"},
                                  undefined,<<>>,dcp_producer_conn,[],
                                  #Port<0.43479>,<0.28205.3>,false}
      ** Reason for termination ==
      ** socket_closed
      =========================CRASH REPORT=========================
        crasher:
          initial call: dcp_proxy:init/1
          pid: <0.28207.3>
          registered_name: []
          exception exit: socket_closed
            in function  gen_server:handle_common_reply/8 (gen_server.erl, line 726)
          ancestors: ['dcp_replicator-default-ns_1@172.23.123.134',
                        'dcp_sup-default','single_bucket_kv_sup-default',
                        ns_bucket_sup,ns_bucket_worker_sup,ns_server_sup,
                        ns_server_nodes_sup,<0.207.0>,ns_server_cluster_sup,
                        root_sup,<0.118.0>]
          message_queue_len: 0
          messages: []
          links: [<0.28204.3>]
          dictionary: []
          trap_exit: false
          status: running
          heap_size: 1598
          stack_size: 27
          reductions: 3747326
        neighbours:
      

      Logs:
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcr/source.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcr/target.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          > panic: interface conversion: *parts.CapiNozzle is not common.OutNozzle: missing method SetUpstreamObjRecycler

          The reason for failure is as mentioned in the panic message. The CapiNozzle does not implement SetUpstreamObjRecycler and while doing runtime type casting (as below) panic happens

          outNozzle := nozzle.(common.OutNozzle)

          This change was introduced in the commit: https://github.com/couchbase/goxdcr/commit/a0fec333b2544f4fde22af99c60eb9929fcd67b0#diff-5828b09aa13d0f510d932422d17b3c244a12a0280a3d43f3ee4a9f6ab4dea9a0

           

          sudeep.jathar Sudeep Jathar added a comment - > panic: interface conversion: *parts.CapiNozzle is not common.OutNozzle: missing method SetUpstreamObjRecycler The reason for failure is as mentioned in the panic message. The CapiNozzle does not implement SetUpstreamObjRecycler and while doing runtime type casting (as below) panic happens outNozzle := nozzle.(common.OutNozzle) This change was introduced in the commit: https://github.com/couchbase/goxdcr/commit/a0fec333b2544f4fde22af99c60eb9929fcd67b0#diff-5828b09aa13d0f510d932422d17b3c244a12a0280a3d43f3ee4a9f6ab4dea9a0  

          Panic is fixed with Sudeep's toybuild. Replication also works as expected.

          pavithra.mahamani Pavithra Mahamani (Inactive) added a comment - Panic is fixed with Sudeep's toybuild. Replication also works as expected.

          Build couchbase-server-6.6.6-10534 contains goxdcr commit 9a0d4ae with commit message:
          MB-52470: CAPI nozzle implements additional method of iface common.Nozzle

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.6-10534 contains goxdcr commit 9a0d4ae with commit message: MB-52470 : CAPI nozzle implements additional method of iface common.Nozzle

          People

            ritam.sharma Ritam Sharma
            pavithra.mahamani Pavithra Mahamani (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty