Details
-
Bug
-
Resolution: Fixed
-
Major
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
Encountered a case in the system testing clusters where due to the core dumps generated by MB-45459, the / partition on which couchbase was installed and /tmp partition resides was out of disk space.
[root@localhost ~]# df -kh
|
Filesystem Size Used Avail Use% Mounted on
|
/dev/mapper/centos-root 31G 31G 20K 100% /
|
devtmpfs 12G 0 12G 0% /dev
|
tmpfs 12G 0 12G 0% /dev/shm
|
tmpfs 12G 1.4G 11G 12% /run
|
tmpfs 12G 0 12G 0% /sys/fs/cgroup
|
/dev/xvdb1 170G 1.8G 169G 2% /data
|
/dev/xvda1 497M 232M 266M 47% /boot
|
tmpfs 2.4G 0 2.4G 0% /run/user/0
|
To collect logs for this issue, tried to run the cbcollect_info cmd specifying the --tmp_dir parameter
/opt/couchbase/bin/cbcollect_info /data/tmp/outputfile.zip --upload-host=uploads.couchbase.com --customer=MB-45459 --tmp-dir /data/tmp/
|
This fails with the following error :
...
|
...
|
Rebalance Report: /opt/couchbase/var/lib/couchbase/logs/rebalance/rebalance_report_2021-04-05T02:47:25Z.json () - OK
|
Rebalance Report: /opt/couchbase/var/lib/couchbase/logs/rebalance/rebalance_report_2021-04-05T04:23:57Z.json () - OK
|
Rebalance Report: /opt/couchbase/var/lib/couchbase/logs/rebalance/rebalance_report_2021-04-05T06:55:15Z.json () - OK
|
Rebalance Report: /opt/couchbase/var/lib/couchbase/logs/rebalance/rebalance_report_2021-04-05T16:55:11Z.json () - OK
|
Rebalance Report: /opt/couchbase/var/lib/couchbase/logs/rebalance/rebalance_report_2021-04-05T20:05:11Z.json () - OK
|
NS Log () - OK
|
Phosphor Trace (['kv_trace_dump', '-H', '127.0.0.1:11209', '-u', '@ns_server', 'kv_trace.json']) - Exit code 1
|
Removing existing snapshot directory '/opt/couchbase/var/lib/couchbase/stats_data/snapshots'
|
Failed to create prometheus snapshot: 7
|
cbcollect_info log () - OK
|
Removing existing snapshot directory '/opt/couchbase/var/lib/couchbase/stats_data/snapshots'
|
Traceback (most recent call last):
|
File "/opt/couchbase/lib/python/cbcollect_info", line 2188, in <module>
|
main()
|
File "/opt/couchbase/lib/python/cbcollect_info", line 2160, in main
|
do_upload_and_exit(zip_filename, upload_url, options.upload_proxy)
|
File "/opt/couchbase/lib/python/cbcollect_info", line 1735, in do_upload_and_exit
|
output_fd, output_file = tempfile.mkstemp()
|
File "/opt/couchbase/lib/python/runtime/lib/python3.9/tempfile.py", line 325, in mkstemp
|
prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
|
File "/opt/couchbase/lib/python/runtime/lib/python3.9/tempfile.py", line 118, in _sanitize_params
|
dir = gettempdir()
|
File "/opt/couchbase/lib/python/runtime/lib/python3.9/tempfile.py", line 287, in gettempdir
|
tempdir = _get_default_tempdir()
|
File "/opt/couchbase/lib/python/runtime/lib/python3.9/tempfile.py", line 219, in _get_default_tempdir
|
raise FileNotFoundError(_errno.ENOENT,
|
FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/tmp']
|
/data/tmp has the correct permissions and ownership as well -
[root@localhost tmp]# ls -la /data/
|
total 4
|
drwxrwxrwx. 5 couchbase couchbase 49 Apr 5 15:22 .
|
dr-xr-xr-x. 19 root root 4096 Oct 28 2019 ..
|
drwxrwxrwx. 2 couchbase couchbase 6 Apr 3 21:06 archive
|
drwxrwxrwx. 14 couchbase couchbase 188 Apr 5 01:54 couchbase
|
drwxrwxrwx. 2 couchbase couchbase 6 Apr 5 17:01 tmp
|