Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30969

[System Test] : Rebalance failed with HYR0115: Local network error

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 6.0.0
    • 6.0.0
    • analytics
    • Centos longevity

    Description

      Build: 6.0.0-1529
      Cycle: 2nd 

      Test : Longevity

      Job: http://qa.sc.couchbase.com/job/centos-systest-launcher/1580/console 

      Cluster : http://172.23.108.103:8091/ 

      Rebalance out of an analytics node ailed with 

      Test Step

      [2018-08-18T11:09:06-07:00, sequoiatools/couchbase-cli:222e63] rebalance -c 172.23.108.103:8091 --server-remove 172.23.96.145 -u Administrator -p password
       
      Error occurred on container - sequoiatools/couchbase-cli:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.96.145 -u Administrator -p password]
       
      docker logs 222e63
      docker start 222e63
       
      *Unable to display progress bar on this os

      Rebalance exited with reason {service_rebalance_failed,cbas,
      {rebalance_failed,
      {service_error,
      <<"Rebalance 7cffef6394a5c9d7b44407f85dd47cfb failed: HYR0115: Local network error">>}}}ns_orchestrator 000ns_1@172.23.108.103 11:17:02 AM   Sat Aug 18, 2018
      

      Error in analytics info logs

      2018-08-18T11:17:01.144-07:00 INFO CBAS.rebalance.Rebalance [Executor-1617:ClusterController] active events handler resumed
      2018-08-18T11:17:01.147-07:00 ERRO CBAS.rebalance.Rebalance [Executor-1617:ClusterController] rebalance failed
      org.apache.hyracks.api.exceptions.HyracksDataException: HYR0115: Local network error
      at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:60) ~[hyracks-api.jar:6.0.0-1529]
      at org.apache.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender(NonDeterministicChannelReader.java:115) ~[hyracks-dataflow-std.jar:6.0.0-1529]
      at org.apache.hyracks.dataflow.std.collectors.NonDeterministicFrameReader.nextFrame(NonDeterministicFrameReader.java:43) ~[hyracks-dataflow-std.jar:6.0.0-1529]
      at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:391) ~[hyracks-control-nc.jar:6.0.0-1529]
      at org.apache.hyracks.control.nc.Task.run(Task.java:330) ~[hyracks-control-nc.jar:6.0.0-1529]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
      at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
      2018-08-18T11:17:01.162-07:00 INFO CBAS.work.JobCleanupWork [Worker:ClusterController] Cleanup for job: JID:0.59339
      2018-08-18T11:17:01.163-07:00 INFO CBAS.work.JobCleanupWork [Worker:ClusterController] Cleanup for job: JID:0.59321
      2018-08-18T11:17:01.167-07:00 INFO CBAS.rebalance.Rebalance [Executor-1617:ClusterController] %%%%%%%%%%%%% exit Rebalance 7cffef6394a5c9d7b44407f85dd47cfb
      2018-08-18T11:17:01.167-07:00 INFO CBAS.runtime.WriteLockCallable [Executor-1617:ClusterController] released lock for Rebalance 7cffef6394a5c9d7b44407f85dd47cfb
      2018-08-18T11:17:01.172-07:00 INFO CBAS.work.JobCleanupWork [Worker:ClusterController] Cleanup for job: JID:0.59331
      2018-08-18T11:17:01.172-07:00 INFO CBAS.work.JobCleanupWork [Worker:ClusterController] Cleanup for job: JID:0.59330
      2018-08-18T11:17:01.172-07:00 INFO CBAS.work.JobCleanupWork [Worker:ClusterController] Cleanup for job: JID:0.59337
      2018-08-18T11:17:01.198-07:00 INFO CBAS.servlet.RebalanceServlet [HttpExecutor(port:9111)-2] Rebalance status request for 7cffef6394a5c9d7b44407f85dd47cfb
      2018-08-18T11:17:01.199-07:00 ERRO CBAS.servlet.RebalanceServlet [HttpExecutor(port:9111)-2] Rebalance 7cffef6394a5c9d7b44407f85dd47cfb failed
      org.apache.hyracks.api.exceptions.HyracksDataException: HYR0115: Local network error
      at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:60) ~[hyracks-api.jar:6.0.0-1529]
      at org.apache.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender(NonDeterministicChannelReader.java:115) ~[hyracks-dataflow-std.jar:6.0.0-1529]
      at org.apache.hyracks.dataflow.std.collectors.NonDeterministicFrameReader.nextFrame(NonDeterministicFrameReader.java:43) ~[hyracks-dataflow-std.jar:6.0.0-1529]
      at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:391) ~[hyracks-control-nc.jar:6.0.0-1529]
      at org.apache.hyracks.control.nc.Task.run(Task.java:330) ~[hyracks-control-nc.jar:6.0.0-1529]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
      at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
      2018-08-18T11:17:01.236-07:00 INFO CBAS.messaging.CCMessageBroker [Executor-1528:ClusterController] Received message: ExecuteStatementRequestMessage(id=6987, from=801fd53f141647b18bf2f6ad1e3743cd): SELECT name as id, result as bucketName, `type` as `Type`, array_length(profile.friends) as num_friends FROM ds2 where duration between 3009 and 3010 and profile is not missing;;

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              vikas.chaudhary Vikas Chaudhary
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty