Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37020

[IPV6]: Bucket op failing with LCB 27

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Build: 6.5.0-4858

      Eventing sanity is failing as bucket op failed with LCB_27 (COUCHBASE_CLIENT_ENOCONF)

      ./testrunner -i /tmp/testexec.14323.ini -p get-cbcollect-info=False,get-cbcollect-info=True -t eventing.eventing_sanity.EventingSanity.test_create_mutation_for_dcp_stream_boundary_from_beginning,nodes_init=4,services_init=kv-eventing-index-n1ql,dataset=default,groups=simple,reset_services=True,skip_cleanup=True
       
      Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 0 Expected : 2016  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'[fd63:6f75:6368:20d4:b164:126b:5258:cac6]': {u'27': 1884}}
      

      Failure summary 

       vikass-MacBook-Pro:~ vikaschaudhary$ python jenkins_log_parser.py http://qa.sc.couchbase.com/job/test_suite_executor/176478
      1	test_create_mutation_for_dcp_stream_boundary_from_beginning
      	Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 0 Expected : 2016  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'[fd63:6f75:6368:20d4:b164:126b:5258:cac6]': {u'27': 1884}}
       
       
      2	test_delete_mutation_for_dcp_stream_boundary_from_beginning
      	Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 0 Expected : 2016  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'[fd63:6f75:6368:20d4:b164:126b:5258:cac6]': {u'27': 1803}}
       
       
      3	test_update_mutation_for_dcp_stream_boundary_from_now
      	Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 0 Expected : 2016  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'[fd63:6f75:6368:20d4:b164:126b:5258:cac6]': {u'27': 2016}}
       
       
      4	test_expiry_mutation_for_dcp_stream_boundary_from_beginning
      	Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 0 Expected : 2016  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'[fd63:6f75:6368:20d4:b164:126b:5258:cac6]': {u'27': 1885}}
       
       
      7	test_doc_timer_events_from_handler_code_with_bucket_ops
      	Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 0 Expected : 2016  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'[fd63:6f75:6368:20d4:b164:126b:5258:cac6]': {u'27': 1802}}
       
       
      9	test_cron_timer_events_from_handler_code_with_bucket_ops
      	Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 0 Expected : 2016  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'[fd63:6f75:6368:20d4:b164:126b:5258:cac6]': {u'27': 1802}}
       
       
      10	test_delete_bucket_operation_from_handler_code
      	Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 0 Expected : 2016  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'[fd63:6f75:6368:20d4:b164:126b:5258:cac6]': {u'27': 1886}}
       
       
      11	test_timers_without_context
      	Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 0 Expected : 2016  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'[fd63:6f75:6368:20d4:b164:126b:5258:cac6]': {u'27': 1773}}
       
       
      12	test_cancel_timers_with_timers_being_overwritten
      	Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 0 Expected : 2016  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'[fd63:6f75:6368:20d4:b164:126b:5258:cac6]': {u'27': 1227}}
       
       
      13	test_source_bucket_mutations_with_timers
      	Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 2016 Expected : 4032  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'[fd63:6f75:6368:20d4:b164:126b:5258:cac6]': {u'27': 1314}}
       
       
      14	test_source_bucket_mutations
      	Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 2016 Expected : 4032  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'[fd63:6f75:6368:20d4:b164:126b:5258:cac6]': {u'27': 1886}}
       
       
      15	test_source_doc_mutations
      	Exception: Eventing has not processed all the mutation in expected time, docs:0  expected doc: 2016
       
       
      16	test_source_doc_mutations_with_timers
      	Exception: Eventing has not processed all the mutation in expected time, docs:0  expected doc: 2016
       
       
      17	test_pause_resume_execution
      	Exception: Eventing took lot of time for handler Function_991174942_test_pause_resume_execution to paused
       
       
      18	test_source_bucket_mutation_for_dcp_stream_boundary_from_now
      	Exception: Eventing has not processed all the dcp_mutation. Current : 2016 Expected : 4032
       
       
      19	test_compress_handler
      	Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 0 Expected : 2016  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'[fd63:6f75:6368:20d4:b164:126b:5258:cac6]': {u'27': 1887}}

      only n1ql op test passed 

      http://qa.sc.couchbase.com/job/test_suite_executor/176478/consoleText

      Attachments

        1. test_1.zip
          10.64 MB
        2. test_19.zip
          47.30 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          vikas.chaudhary Vikas Chaudhary created issue -
          jeelan.poola Jeelan Poola made changes -
          Field Original Value New Value
          Assignee Jeelan Poola [ jeelan.poola ] Suraj Naik [ suraj.naik ]

          Bucket op failing only on IPv6 cluster.

          Eventing log shows:

          2019-11-25T16:38:44.008+05:30 [Info] eventing-consumer [worker_test_2:/tmp/[::1]:9000_2_3746722029.sock:19776] [lcb,connection L:156 I:4186126171] <[::1:9000]:8091> (SOCK=ef3d9144d42eda7f) Failed to establish connection: LCB_ENETUNREACH (0x31), os errno=101
          2019-11-25T16:38:44.008+05:30 [Info] eventing-consumer [worker_test_2:/tmp/[::1]:9000_2_3746722029.sock:19776] [lcb,htconfig L:301 I:4186126171] Connection to REST API failed with LCB_ENETUNREACH (0x31) (os errno = 101)
          2019-11-25T16:38:44.008+05:30 [Info] eventing-consumer [worker_test_2:/tmp/[::1]:9000_2_3746722029.sock:19776] [lcb,confmon L:179 I:4186126171] Provider 'HTTP' failed
          2019-11-25T16:38:44.008+05:30 [Info] eventing-consumer [worker_test_2:/tmp/[::1]:9000_2_3746722029.sock:19776] [lcb,bootstrap L:170 I:4186126171] Failed to bootstrap client=0x7fa4f00cd680. Error=LCB_ENETUNREACH (0x31), Message=No more bootstrap providers remain
          

          Eventing application log shows:

          2019-11-25T16:42:19.924+05:30 [INFO] {"message":{"code":8,"desc":"Client not bootstrapped. Ensure bootstrap/connect was attempted and was successful","name":"LCB_CLIENT_ENOCONF"}
          

          I observed that this happens after the call to * lcb_wait* fails. We do retry 5 times from the eventing side but still we are not able to get the connection.

          suraj.naik Suraj Naik (Inactive) added a comment - Bucket op failing only on IPv6 cluster. Eventing log shows: 2019-11-25T16:38:44.008+05:30 [Info] eventing-consumer [worker_test_2:/tmp/[::1]:9000_2_3746722029.sock:19776] [lcb,connection L:156 I:4186126171] <[::1:9000]:8091> (SOCK=ef3d9144d42eda7f) Failed to establish connection: LCB_ENETUNREACH (0x31), os errno=101 2019-11-25T16:38:44.008+05:30 [Info] eventing-consumer [worker_test_2:/tmp/[::1]:9000_2_3746722029.sock:19776] [lcb,htconfig L:301 I:4186126171] Connection to REST API failed with LCB_ENETUNREACH (0x31) (os errno = 101) 2019-11-25T16:38:44.008+05:30 [Info] eventing-consumer [worker_test_2:/tmp/[::1]:9000_2_3746722029.sock:19776] [lcb,confmon L:179 I:4186126171] Provider 'HTTP' failed 2019-11-25T16:38:44.008+05:30 [Info] eventing-consumer [worker_test_2:/tmp/[::1]:9000_2_3746722029.sock:19776] [lcb,bootstrap L:170 I:4186126171] Failed to bootstrap client=0x7fa4f00cd680. Error=LCB_ENETUNREACH (0x31), Message=No more bootstrap providers remain Eventing application log shows: 2019-11-25T16:42:19.924+05:30 [INFO] {"message":{"code":8,"desc":"Client not bootstrapped. Ensure bootstrap/connect was attempted and was successful","name":"LCB_CLIENT_ENOCONF"} I observed that this happens after the call to * lcb_wait* fails. We do retry 5 times from the eventing side but still we are not able to get the connection.
          suraj.naik Suraj Naik (Inactive) made changes -
          Assignee Suraj Naik [ suraj.naik ] Jeelan Poola [ jeelan.poola ]
          jeelan.poola Jeelan Poola added a comment - - edited

          Matt Ingenthron Request someone from clients team to take a look. Please assign it back if eventing needs to take care of something here. Thank you!

          jeelan.poola Jeelan Poola added a comment - - edited Matt Ingenthron Request someone from clients team to take a look. Please assign it back if eventing needs to take care of something here. Thank you!
          jeelan.poola Jeelan Poola made changes -
          Component/s clients [ 10042 ]
          jeelan.poola Jeelan Poola made changes -
          Component/s eventing [ 14026 ]

          Jeelan Poola: current lcb requires a parameter to be passed if using IPv6 (and this is planned to change in 3.0). Can you tell me what the parameters used in this case are?

          See the docs on IPv6.

          I'll ask someone else to have a look, but I think in this case it'll be the responsibility of the initializer to turn on IPv6 with the current API, not just pass an address that looks up as AAAA or is IPv6 format.

          ingenthr Matt Ingenthron added a comment - Jeelan Poola : current lcb requires a parameter to be passed if using IPv6 (and this is planned to change in 3.0). Can you tell me what the parameters used in this case are? See the docs on IPv6 . I'll ask someone else to have a look, but I think in this case it'll be the responsibility of the initializer to turn on IPv6 with the current API, not just pass an address that looks up as AAAA or is IPv6 format.
          ingenthr Matt Ingenthron made changes -
          Assignee Jeelan Poola [ jeelan.poola ] Sergey Avseyev [ avsej ]

          [::1:9000]:8091 this address does not look valid. You have to specify it like this: [::1]:9000, also make sure that `ipv6=allow` specified in the connection string

          avsej Sergey Avseyev added a comment - [::1:9000] :8091 this address does not look valid. You have to specify it like this: [::1] :9000 , also make sure that `ipv6=allow` specified in the connection string
          avsej Sergey Avseyev made changes -
          Assignee Sergey Avseyev [ avsej ] Vikas Chaudhary [ vikas.chaudhary ]
          lynn.straus Lynn Straus made changes -
          Labels ipv6 approved-for-mad-hatter ipv6
          lynn.straus Lynn Straus made changes -
          Due Date 26/Nov/19
          ritam.sharma Ritam Sharma made changes -
          Assignee Vikas Chaudhary [ vikas.chaudhary ] Jeelan Poola [ jeelan.poola ]
          jeelan.poola Jeelan Poola made changes -
          Assignee Jeelan Poola [ jeelan.poola ] Suraj Naik [ suraj.naik ]
          lynn.straus Lynn Straus made changes -
          Link This issue blocks MB-36676 [ MB-36676 ]
          suraj.naik Suraj Naik (Inactive) made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          suraj.naik Suraj Naik (Inactive) added a comment - - edited

          Sergey Avseyev

          Thanks for the observation, we formed the address correctly for both source bucket and meta data bucket. But the binding bucket's addresses were malformed. Have uploaded the patch to the fix the same.
          Hence, changing the component to eventing.

          suraj.naik Suraj Naik (Inactive) added a comment - - edited Sergey Avseyev Thanks for the observation, we formed the address correctly for both source bucket and meta data bucket. But the binding bucket's addresses were malformed. Have uploaded the patch to the fix the same. Hence, changing the component to eventing.
          suraj.naik Suraj Naik (Inactive) made changes -
          Component/s eventing [ 14026 ]
          Component/s clients [ 10042 ]
          jeelan.poola Jeelan Poola made changes -
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Resolved [ 5 ]

          Build couchbase-server-6.5.0-4893 contains eventing commit 24bab97 with commit message:
          MB-37040 MB-37020 Merge remote-tracking branch 'couchbase/unstable' into

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4893 contains eventing commit 24bab97 with commit message: MB-37040 MB-37020 Merge remote-tracking branch 'couchbase/unstable' into

          Build couchbase-server-6.5.0-4893 contains eventing commit 7d08239 with commit message:
          MB-37020: Fix bucket get failure for IPv6 cluster

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4893 contains eventing commit 7d08239 with commit message: MB-37020 : Fix bucket get failure for IPv6 cluster

          Build couchbase-server-7.0.0-1080 contains eventing commit 24bab97 with commit message:
          MB-37040 MB-37020 Merge remote-tracking branch 'couchbase/unstable' into

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1080 contains eventing commit 24bab97 with commit message: MB-37040 MB-37020 Merge remote-tracking branch 'couchbase/unstable' into

          Build couchbase-server-7.0.0-1080 contains eventing commit 7d08239 with commit message:
          MB-37020: Fix bucket get failure for IPv6 cluster

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1080 contains eventing commit 7d08239 with commit message: MB-37020 : Fix bucket get failure for IPv6 cluster

          verified on 6.5.0-4893

          vikas.chaudhary Vikas Chaudhary added a comment - verified on 6.5.0-4893
          vikas.chaudhary Vikas Chaudhary made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            suraj.naik Suraj Naik (Inactive)
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty