Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50780

[Upgrade] Handler underwent automatic undeployment after upgrade to Neo

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • 7.1.0
    • eventing
    • 6.6.5-10080
      7.1.0-2179
    • Untriaged
    • Centos 64-bit
    • 1
    • Unknown

    Description

      QE TEST

      ./testrunner -i /tmp/win10-bucket-ops.ini -p  -t eventing.eventing_upgrade.EventingUpgrade.test_offline_upgrade_with_eventing,nodes_init=5,dataset=default,groups=simple,upgrade_test=True,initial_version=6.6.5-10080,doc-per-day=10,upgrade_version=7.1.0-2179,get-cbcollect-info=True

      STEPS

      1. Cluster consists of 5 nodes (2 kv, 1 eventing, index and query node each) running on CB Server 6.6.5-10080.
      2. Create 2 eventing handlers - bucket_op and timers.
      3. Deploy timers handler.
      4. Perform offline upgrade of the cluster to 7.1.0-2179.

      OBSERVATION
      timers handler underwent auto undeployment during upgrade.

      Error msgs on eventing node -

      grep "\[Error\]" ns_server.eventing.log             
      2022-02-03T03:37:36.853-08:00 [Error] util::ReadAppContent Function: bucket_op app content doesn't exist or is empty
      		 
      2022-02-03T03:37:37.973-08:00 [Error] util::ReadAppContent Function: timers app content doesn't exist or is empty
      		 
      2022-02-03T03:38:39.192-08:00 [Error] [gocb] memdClient read failure: EOF
      		 
      2022-02-03T03:38:39.478-08:00 [Error] feed.DcpGetSeqnos(): EOF
      		 
      2022-02-03T03:38:39.478-08:00 [Error] Consumer::dcpEventsRemainingToProcess [worker_timers_0:/tmp/127.0.0.1:8091_0_751876559.sock:47628] Failed to fetch get_all_vb_seqnos, err: EOF
      		 
      2022-02-03T03:38:39.528-08:00 [Error] [gocb] memdClient read failure: read tcp 172.23.106.73:50866->172.23.106.64:11210: read: connection reset by peer
      		 
      2022-02-03T03:38:39.583-08:00 [Error] StartDcpFeedOver(): dial tcp 172.23.106.64:11210: connect: connection refused
      		 
      2022-02-03T03:38:39.583-08:00 [Error] Consumer::dcpEventsRemainingToProcess [worker_timers_1:/tmp/127.0.0.1:8091_1_751876559.sock:47636] Failed to fetch get_all_vb_seqnos, err: dial tcp 172.23.106.64:11210: connect: connection refused
      		 
      2022-02-03T03:38:39.659-08:00 [Error] StartDcpFeedOver(): dial tcp 172.23.106.64:11210: connect: connection refused
      		 
      2022-02-03T03:38:39.659-08:00 [Error] Consumer::dcpEventsRemainingToProcess [worker_timers_2:/tmp/127.0.0.1:8091_2_751876559.sock:47652] Failed to fetch get_all_vb_seqnos, err: dial tcp 172.23.106.64:11210: connect: connection refused
      		 
      2022-02-03T03:39:57.579-08:00 [Error] Supervisor::gocbConnectBucket Failed to connect to bucket metadata, err: unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"WaitUntilReady","Opaque":"","TimeObserved":5000714969,"RetryReasons":["NOT_READY","CONNECTION_ERROR"],"RetryAttempts":18,"LastDispatchedTo":"","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2022-02-03T03:40:03.581-08:00 [Error] Supervisor::gocbConnectBucket Failed to connect to bucket metadata, err: unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"WaitUntilReady","Opaque":"","TimeObserved":5000372980,"RetryReasons":["CONNECTION_ERROR"],"RetryAttempts":18,"LastDispatchedTo":"","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2022-02-03T03:40:09.581-08:00 [Error] Supervisor::gocbConnectBucket Failed to connect to bucket metadata, err: unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"WaitUntilReady","Opaque":"","TimeObserved":5000130003,"RetryReasons":["CONNECTION_ERROR"],"RetryAttempts":18,"LastDispatchedTo":"","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2022-02-03T03:40:15.582-08:00 [Error] Supervisor::gocbConnectBucket Failed to connect to bucket metadata, err: unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"WaitUntilReady","Opaque":"","TimeObserved":5000126809,"RetryReasons":["CONNECTION_ERROR"],"RetryAttempts":18,"LastDispatchedTo":"","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2022-02-03T03:40:21.584-08:00 [Error] Supervisor::gocbConnectBucket Failed to connect to bucket metadata, err: unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"WaitUntilReady","Opaque":"","TimeObserved":5000813421,"RetryReasons":["CONNECTION_ERROR"],"RetryAttempts":18,"LastDispatchedTo":"","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2022-02-03T03:40:27.585-08:00 [Error] Supervisor::gocbConnectBucket Failed to connect to bucket metadata, err: unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"WaitUntilReady","Opaque":"","TimeObserved":5000843316,"RetryReasons":["CONNECTION_ERROR"],"RetryAttempts":18,"LastDispatchedTo":"","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2022-02-03T03:40:33.586-08:00 [Error] Supervisor::gocbConnectBucket Failed to connect to bucket metadata, err: unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"WaitUntilReady","Opaque":"","TimeObserved":5000065480,"RetryReasons":["CONNECTION_ERROR"],"RetryAttempts":18,"LastDispatchedTo":"","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2022-02-03T03:40:39.586-08:00 [Error] Supervisor::gocbConnectBucket Failed to connect to bucket metadata, err: unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"WaitUntilReady","Opaque":"","TimeObserved":5000085063,"RetryReasons":["CONNECTION_ERROR"],"RetryAttempts":18,"LastDispatchedTo":"","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2022-02-03T03:40:45.992-08:00 [Error] Supervisor::gocbConnectBucket Failed to connect to bucket metadata, err: unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"WaitUntilReady","Opaque":"","TimeObserved":5000781669,"RetryReasons":["CONNECTION_ERROR"],"RetryAttempts":18,"LastDispatchedTo":"","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2022-02-03T03:41:08.550-08:00 [Error] SuperSupervisor::GetDcpEventsRemainingToProcess [0] Function: timers request didn't go through as Eventing.Producer instance isn't alive
      		 
      2022-02-03T03:41:14.031-08:00 [Error] Producer::handleV8Consumer [timers:0] Accept failed in main loop, err: accept unix /tmp/127.0.0.1:8091_0_751876559.sock: use of closed network connection
      		 
      2022-02-03T03:41:14.031-08:00 [Error] Producer::handleV8Consumer [timers:0] Accept failed in main loop, err: accept unix /tmp/127.0.0.1:8091_1_751876559.sock: use of closed network connection
      		 
      2022-02-03T03:41:14.031-08:00 [Error] Producer::handleV8Consumer [timers:0] Accept failed in main loop, err: accept unix /tmp/127.0.0.1:8091_2_751876559.sock: use of closed network connection
      		 
      2022-02-03T03:41:14.031-08:00 [Error] Producer::handleV8Consumer [timers:0] Accept failed in feedback loop, err: accept unix /tmp/f_127.0.0.1:8091_2_751876559.sock: use of closed network connection
      		 
      2022-02-03T03:41:14.032-08:00 [Error] Producer::handleV8Consumer [timers:0] Accept failed in feedback loop, err: accept unix /tmp/f_127.0.0.1:8091_0_751876559.sock: use of closed network connection
      		 
      2022-02-03T03:41:14.032-08:00 [Error] Producer::handleV8Consumer [timers:0] Accept failed in feedback loop, err: accept unix /tmp/f_127.0.0.1:8091_1_751876559.sock: use of closed network connection
      		 
      2022-02-03T03:51:54.269-08:00 [Error] SuperSupervisor::NotifyPrepareTopologyChange [0] 0 eventing nodes supplied as keepNodes

      NOTE
      TLS is disabled for entire duration of this test.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sujay.gad Sujay Gad
            sujay.gad Sujay Gad
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty