Details
-
Bug
-
Resolution: Fixed
-
Major
-
4.5.0
-
None
-
Triaged
-
Unknown
Description
I'm running Couchbase Server 4.5.0 community on an EC2 instance running CentOS7.3 and seeing sporadic startup failures.
Steps to repro
- Login to AWS
- Go to Cloudformation, and choose "Create Stack"
- Check "Specify an Amazon S3 template URL" radio button
- Enter URL: http://cbmobile-bucket.s3.amazonaws.com/issues/issue_mb_23081/sgautoscale_cloudformation_template.json
- Add an appropriate stack name and use a key name that will allow you to ssh into instances
- Click Next twice (no need to customize options)
- Check the checkbox next to "I acknowledge that AWS CloudFormation might create IAM resources." and hit "Create"
It should be very easy to repro with these steps. In my last 2 attempts, I repro'd one out of two times.
Observed Behavior
After the ec2 instance started up, I ssh'd into the machine and ran:
systemctl status couchbase-server
and it shows:
systemctl status couchbase-server
● couchbase-server.service - Couchbase Server
Loaded: loaded (/usr/lib/systemd/system/couchbase-server.service; enabled; vendor preset: disabled)
Active: failed (Result: timeout) since Wed 2017-03-01 05:04:33 UTC; 3min 53s ago
Docs: http://docs.couchbase.com
Mar 01 05:02:56 ip-172-31-12-251 systemd[1]: Starting Couchbase Server...
Mar 01 05:02:59 ip-172-31-12-251 systemd[1]: PID file /opt/couchbase/var/lib/couchbase/couchbase-server.pid not readable (yet?) after start.
Mar 01 05:04:33 ip-172-31-12-251 systemd[1]: couchbase-server.service start operation timed out. Terminating.
Mar 01 05:04:33 ip-172-31-12-251 systemd[1]: Failed to start Couchbase Server.
Mar 01 05:04:33 ip-172-31-12-251 systemd[1]: Unit couchbase-server.service entered failed state.
Mar 01 05:04:33 ip-172-31-12-251 systemd[1]: couchbase-server.service failed.
I was able to restart it successfully:
systemctl status couchbase-server
● couchbase-server.service - Couchbase Server
Loaded: loaded (/usr/lib/systemd/system/couchbase-server.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2017-03-01 05:10:08 UTC; 10s ago
Docs: http://docs.couchbase.com
Process: 2720 ExecStart=/opt/couchbase/bin/couchbase-server – -noinput -detached (code=exited, status=0/SUCCESS)
Main PID: 2800 (beam.smp)
CGroup: /system.slice/couchbase-server.service
├─2732 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/epmd -daemon
├─2769 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 – -root /opt/couchbase/lib/erlang -progname erl – -home /opt/couchbase – -smp enable -kernel ine...
├─2800 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 – -root /opt/couchbase/lib/erlang -progname e...
├─2834 sh -s disksup
├─2836 /opt/couchbase/lib/erlang/lib/os_mon-2.2.14/priv/bin/memsup
├─2837 /opt/couchbase/lib/erlang/lib/os_mon-2.2.14/priv/bin/cpu_sup
├─2838 inet_gethost 4
├─2839 inet_gethost 4
├─2840 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -P 327680 -K true – -root /opt/couchbase/lib/erlang -progname erl – -home /opt/couchbase – -pa /opt/co...
├─2871 sh -s disksup
├─2872 /opt/couchbase/lib/erlang/lib/os_mon-2.2.14/priv/bin/memsup
├─2874 /opt/couchbase/lib/erlang/lib/os_mon-2.2.14/priv/bin/cpu_sup
├─2880 /opt/couchbase/bin/priv/godu
├─2885 sh -s ns_disksup
├─2886 /opt/couchbase/bin/priv/godu
├─2892 /opt/couchbase/bin/saslauthd-port
├─2897 portsigar for ns_1@127.0.0.1
├─2898 /opt/couchbase/bin/goport
├─2901 /opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
└─2904 /opt/couchbase/bin/goxdcr -sourceKVAdminPort=8091 -xdcrRestPort=9998 -isEnterprise=false
Mar 01 05:10:04 ip-172-31-12-251 systemd[1]: Starting Couchbase Server...
Mar 01 05:10:05 ip-172-31-12-251 systemd[1]: PID file /opt/couchbase/var/lib/couchbase/couchbase-server.pid not readable (yet?) after start.
Mar 01 05:10:08 ip-172-31-12-251 systemd[1]: couchbase-server.service: Supervising process 2800 which is not our child. We'll most likely not notice when it exits.
Mar 01 05:10:08 ip-172-31-12-251 systemd[1]: Started Couchbase Server.