Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-21367

Rebalancing with couchbase-cli causes "close failed in file object destructor: sys.excepthook is missing; lost sys.stderr"

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 5.0.0
    • 4.0.0
    • tools
    • None
    • Ubuntu 14.04 under AWS, when deploying with Juju.
    • Untriaged
    • Ubuntu 64-bit
    • CBCollectInfo attached.
    • No

    Description

      I get two couchbase servers ready. Server A and server B. Server A is the master node, which was installed first. Server B was installed later and is instructed to join the cluster, by issuing the 'rebalance' command. However, this happens:

      First try:

          INFO cluster-relation-changed + /opt/couchbase/bin/couchbase-cli rebalance -u juju -p charmtest -c 172.31.23.65 --server-add=172.31.5.101 --server-add-username=juju --server-add-password=charmtest --service=data,index,query
          INFO cluster-relation-changed ERROR: unable to server-add 172.31.5.101:8091 (400) Bad Request
          INFO cluster-relation-changed ["Prepare join failed. Could not connect to \"172.31.5.101\" on port 8091.  This could be due to an incorrect host/port combination or a firewall in place between the servers."]
          ERROR juju.worker.uniter.operation runhook.go:107 hook "cluster-relation-changed" failed: exit status 2
       
       
      It goes like this for a couple times. Then, this happens (here's the weird part):
       
          INFO cluster-relation-changed + /opt/couchbase/bin/couchbase-cli rebalance -u juju -p charmtest -c 172.31.23.65 --server-add=172.31.5.101 --server-add-username=juju --server-add-password=charmtest --service=data,index,query
          INFO cluster-relation-changed INFO: rebalancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
          INFO cluster-relation-changed SUCCESS: rebalanced cluster
          INFO cluster-relation-changed close failed in file object destructor:
          INFO cluster-relation-changed sys.excepthook is missing
          INFO cluster-relation-changed lost sys.stderr
      

      I don't know why this error is coming up. After that, any future attempts would get this:

          INFO cluster-relation-changed ERROR: unable to server-add 172.31.5.101:8091 (400) Bad Request
          INFO cluster-relation-changed ["Prepare join failed. Node is already part of cluster."]
      

      So, couchbase is in fact rebalancing the cluster with the node. However, for some weird reason, it's failing. All of these logs are from server B.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mikew Mike Wiederhold [X] (Inactive)
            joseeantonior José Antonio Rey
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty