Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30978

[System test]: Unable to find given hostport

    XMLWordPrintable

Details

    • Bug
    • Resolution: Incomplete
    • Major
    • 6.0.0
    • 6.0.0
    • analytics
    • Centos longevity

    Description

      Build:  6.0.0 build 1529

      Test Job: http://qa.sc.couchbase.com/job/centos-systest-launcher/1580/console 

      Cluster: http://172.23.108.103:8091/ 

      We run the following steps in centos longevity 

      Longevity :

      1. Create 22 node cluster  (9 KV,5 index, 2 query , 2 fts, 2 eventing, 2 cbas)
      2. Create 10 buckets (default bucket with Active compression)
      3. Create views
      4. Load data
      5. Remove kv node
      6. Deploy eventing functions
      7. Create dataset on analytics on 4 buckets
      8. Create index on 2 datasets
      9. Create 2i index
      10. Load more data
      11. Run queries on 2i
      12. Swap a KV node
      13. Run 240 queries per second on analytics
      14. Connect link Local
      15. Load more data to default bucket
      16. Add eventing node
      17. Remove eventing node
      18. Swap eventing node
      19. Disconnect link Local 
      20. Add analytics node 
      21. connect link Local
      22. Disconnect link Local 
      23. Remove analytics node
      24. connect link Local
      25. Swap analytics node
      26. Kill analytics nodes
      27. Run views
      28. Create fts indexes 
      29. Regex search on FTS
      30. XDCR replication
      31. Add rbac users
      32. Undeploy eventing handlers
      33. Load 1M doc
      34. Create 2i indexes 
      35. Rebalance in index
      36. Rebalance out index
      37. Swap index node
      38. Rebalance in 2 index nodes
      39. Rebalance out 2 index nodes
      40. Rebalance out 1 KV
      41. Rebalance in 1 KV
      42. Failover -> Full recovery index node
      43. Failover -> Rebalance out index node
      44. Add index node
      45. Redeploy eventing handlers
      46. Run Tpcc
      47. Update Doc
      48. Add a kv node , failover kv node -> rebalance 
      49. swap hard failover -> Add 1 KV remove 2 KV as soft and hard failover
      50. Multinode autofailover -> failover 3 KV nodes and rebalance 

      Observed the following error during test 

      Test step

      [2018-08-18T11:35:17-07:00, sequoiatools/couchbase-cli:2f31b3] server-add -c 172.23.108.103:8091 --server-add 172.23.104.164:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services analytics
      		 
      [2018-08-18T11:35:43-07:00, sequoiatools/couchbase-cli:1468d8] rebalance -c 172.23.108.103:8091 --server-remove 172.23.96.145 -u Administrator -p password
      		 
      [2018-08-18T12:39:59-07:00, sequoiatools/cmd:7bc147] 60 -- sleep
      		 
      [2018-08-18T12:41:21-07:00, sequoiatools/cmd:303817] 300 -- sleep
      		 
      [2018-08-18T12:55:38-07:00, sequoiatools/cmd:4ac8c9] 180 -- sleep

      Errors

      172.23.96.145
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-08-17T06:44:42.633-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:0.35436:TAID:TID:ANID:ODID:2:0:5:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-08-17T06:44:44.010-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:0.35440:TAID:TID:ANID:ODID:2:0:6:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-08-17T06:44:44.071-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:0.35440:TAID:TID:ANID:ODID:2:0:4:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-08-17T06:44:44.107-07:00 FATA CBAS.util.ExitUtil [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:0.35436:TAID:TID:ANID:ODID:2:0:5:0:0] JVM halting with status 103; thread dump at halt: 
      /opt/couchbase/var/lib/couchbase/logs/analytics_info.log:2018-08-18T12:36:19.326-07:00 FATA CBAS.cbas error cbauth.SetRequestAuth(): Unable to find given hostport in cbauth database: `172.23.96.145:9110'
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_shutdown.log:2018-08-18T12:36:17.952-07:00 FATA CBAS.util.ExitUtil [Stdin Watcher] JVM halting with status 33; thread dump at halt: 
      /opt/couchbase/var/lib/couchbase/logs/analytics_warn.log:2018-08-17T06:44:42.633-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:0.35436:TAID:TID:ANID:ODID:2:0:5:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_warn.log:2018-08-17T06:44:44.010-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:0.35440:TAID:TID:ANID:ODID:2:0:6:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_warn.log:2018-08-17T06:44:44.071-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:0.35440:TAID:TID:ANID:ODID:2:0:4:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_warn.log:2018-08-17T06:44:44.107-07:00 FATA CBAS.util.ExitUtil [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:0.35436:TAID:TID:ANID:ODID:2:0:5:0:0] JVM halting with status 103; thread dump at halt: 
      /opt/couchbase/var/lib/couchbase/logs/babysitter.log: {<<"2018-08-18T12:36:19.326-07:00 FATA CBAS.cbas error cbauth.SetRequestAuth(): Unable to find given hostport in cbauth database: `172.23.96.145:9110'\n">>,
      		 
      /opt/couchbase/var/lib/couchbase/logs/debug.log.5.gz:2018-08-18T12:36:19.326-07:00 FATA CBAS.cbas error cbauth.SetRequestAuth(): Unable to find given hostport in cbauth database: `172.23.96.145:9110'
      		 
      /opt/couchbase/var/lib/couchbase/logs/info.log:2018-08-18T12:36:19.326-07:00 FATA CBAS.cbas error cbauth.SetRequestAuth(): Unable to find given hostport in cbauth database: `172.23.96.145:9110'
      		 
       
      		 
      172.23.106.188
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-08-17T06:30:56.992-07:00 FATA CBAS.nc.CcConnection [Executor-2449:fe60c856b10e78da3be6a95a308f59cc] Registering with ClusterControllerRemoteProxy [/172.23.96.145:9112] failed with exception
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-08-17T06:30:58.439-07:00 FATA CBAS.util.ExitUtil [Executor-2449:fe60c856b10e78da3be6a95a308f59cc] JVM halting with status 15; thread dump at halt: 
      /opt/couchbase/var/lib/couchbase/logs/analytics_info.log:2018-08-17T06:30:56.992-07:00 FATA CBAS.nc.CcConnection [Executor-2449:fe60c856b10e78da3be6a95a308f59cc] Registering with ClusterControllerRemoteProxy [/172.23.96.145:9112] failed with exception
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_info.log:2018-08-17T06:30:58.439-07:00 FATA CBAS.util.ExitUtil [Executor-2449:fe60c856b10e78da3be6a95a308f59cc] JVM halting with status 15; thread dump at halt: 
      /opt/couchbase/var/lib/couchbase/logs/analytics_shutdown.log:2018-08-17T06:31:04.016-07:00 FATA CBAS.util.ExitUtil [Stdin Watcher] JVM halting with status 33; thread dump at halt: 
      /opt/couchbase/var/lib/couchbase/logs/analytics_warn.log:2018-08-17T06:30:56.992-07:00 FATA CBAS.nc.CcConnection [Executor-2449:fe60c856b10e78da3be6a95a308f59cc] Registering with ClusterControllerRemoteProxy [/172.23.96.145:9112] failed with exception
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_warn.log:2018-08-17T06:30:58.439-07:00 FATA CBAS.util.ExitUtil [Executor-2449:fe60c856b10e78da3be6a95a308f59cc] JVM halting with status 15; thread dump at halt: 
       
      		 
      172.23.99.25
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_info.log:2018-08-18T12:49:59.292-07:00 FATA CBAS.cbas error cbauth.SetRequestAuth(): Unable to find given hostport in cbauth database: `172.23.99.25:9110'
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_shutdown.log:2018-08-17T05:55:55.428-07:00 FATA CBAS.util.ExitUtil [Stdin Watcher] JVM halting with status 33; thread dump at halt: 
      /opt/couchbase/var/lib/couchbase/logs/analytics_shutdown.log:2018-08-17T06:44:39.621-07:00 FATA CBAS.util.ExitUtil [Stdin Watcher] JVM halting with status 33; thread dump at halt: 
      /opt/couchbase/var/lib/couchbase/logs/analytics_shutdown.log:2018-08-18T12:50:01.027-07:00 FATA CBAS.util.ExitUtil [Stdin Watcher] JVM halting with status 33; thread dump at halt: 
      /opt/couchbase/var/lib/couchbase/logs/debug.log.3.gz:2018-08-18T12:49:59.292-07:00 FATA CBAS.cbas error cbauth.SetRequestAuth(): Unable to find given hostport in cbauth database: `172.23.99.25:9110'
      		 
      /opt/couchbase/var/lib/couchbase/logs/info.log:2018-08-18T12:49:59.292-07:00 FATA CBAS.cbas error cbauth.SetRequestAuth(): Unable to find given hostport in cbauth database: `172.23.99.25:9110'

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-30976 FATAL Errors during system test

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              vikas.chaudhary Vikas Chaudhary
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty