Details
-
Bug
-
Resolution: Incomplete
-
Major
-
None
-
2.0
-
None
-
Security Level: Public
-
None
-
OS: Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-31-generic x86_64)
HW: 8 Core Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz, 16 GB Ram (VMware)
Usage: LOW (this is presently in development, so there is almost no load)
3 - Nodes
Description
I've operated couchbase 2.0 Beta for several months with no issue. I decided to upgrade the 3 node cluster from Beta to Enterprise 2.0 GA for testing.
I used the following procedure to upgrade:
First I removed a node from the cluster, performed a rebalance, then uninstalled Couchbase on the node using the following method:
dpkg -r couchbase-server
dpkg --purge couchbase-server
rm -R /opt/couchbase
dpkg -i couchbase.....
Then I add the node back to the cluster, perform a rebalance, and repeat this process for the other two nodes.
Everything ran ok for a few days, then this morning I received several CB failover emails. Upon logging in I found the following errors on each host:
Jan 22 14:15:04 couch01 kernel: [411073.960652] do_general_protection: 24 callbacks suppressed
Jan 22 14:15:04 couch01 kernel: [411073.960658] beam.smp[1918] general protection ip:7f4bfcabf8c3 sp:7f4c0c2d15e8 error:0 in libstdc++.so.6.0.16[7f4bfca62000+e2000]
Jan 22 14:17:52 couch02 kernel: [1496388.967379] do_general_protection: 12 callbacks suppressed
Jan 22 14:17:52 couch02 kernel: [1496388.967384] beam.smp[10934] general protection ip:7f5f61d548b0 sp:7f5f795665e8 error:0 in libstdc++.so.6.0.16[7f5f61cf7000+e2000]
Jan 22 14:12:17 couch03 kernel: [1496631.103918] beam.smp[24037] general protection ip:7f8a3864b8b0 sp:7f8a431fe5e8 error:0 in libstdc++.so.6.0.16[7f8a385ee000+e2000]
After rebooting the servers at the recommendation of one of the devs (ingenthr). Couchbase would no longer boot, and would crash leaving the following PS running:
1989 ? S 0:00 /opt/couchbase/lib/erlang/erts-5.8.5/bin/epmd -daemon