Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
7.0.1
-
Untriaged
-
1
-
Yes
Description
I found some oddity that shows up when trying to create a backport for MB-47380 into 7.0.1.
MB-47380 has been fine and it still works fine in master.
However, once cherry-picked the portions to CC, re-running cbcollect (without redaction) ends up with empty files that MB-47380 added... namely, goxdcr_remote_clusters.json and goxdcr_replications.json.
Spoke with Steve Watanabe, and he pointed me to a few check-ins lately that also touched cbcollect_info. Namely:
http://review.couchbase.org/c/ns_server/+/157902
and
http://review.couchbase.org/c/ns_server/+/157903
Interestingly, when I reverted http://review.couchbase.org/c/ns_server/+/157902 patch locally, the cbcollect works, which means that the newly added XDCR files have content in them.
It also seems like the same situation of 0 byte files in the zip file also occurs to at least one other file. One other example I found used a similar syntax as the new goxdcr files being generated. Specifically, the entry:
make_curl_task(name="master events",
|
user="@",
|
password=memcached_pass,
|
timeout=300,
|
url='http://%s:%s/diag/masterEvents?o=1' % (
|
local_url_addr, read_guts(guts, "rest_port")),
|
log_file="master_events.log",
|
no_header=True),
|
(And another non-text file, projector_cprof.log)
Ultimately, without reverting the patch, the 2 goxdcr files + the master_events.log + projector_cprof.log are empty in the zip file. And, reverting the patch seems to flush them just fine (non-0 bytes and with valid data). Thus, I don't think it is specific only to goxdcr portion of the log collection.
Visually, I think the patch looks fine, and atm I don't see anything wrong with it.
What's weird is also the fact that all 3 patches (the 2 mentioned above + goxdcr's portion) are present in master, and cbcollect_info runs just fine in master.
File size was a thought, but there are other small files that were flushed just fine with the changeset.
File descriptor count doesn't seem to be a problem, as ulimit -n is big enough on my machine.
Attachments
Issue Links
- relates to
-
-
- Closed
-
Activity
| Assignee | Aliaksey Artamonau [ aliaksey artamonau ] | Steve Watanabe [ steve.watanabe ] |
Without the two recent cbcollect_info changes there is content in the cprof file
$ ll *cprof*
|
-rw-r--r-- 1 stevewatanabe staff 16 Jul 27 09:38 analytics_cprof.log
|
-rw-r--r-- 1 stevewatanabe staff 543 Jul 27 09:38 eventing_cprof.log
|
-rw-r--r-- 1 stevewatanabe staff 1323 Jul 27 09:39 goxdcr_cprof.log
|
-rw-r--r-- 1 stevewatanabe staff 2830 Jul 27 09:37 indexer_cprof.log
|
-rw-r--r-- 1 stevewatanabe staff 107 Jul 27 09:38 projector_cprof.log
|
-rw-r--r-- 1 stevewatanabe staff 1604 Jul 27 09:37 query_cprof.log
|
seems to be related to log files with no_header=True
The problem is that we redact/zip files that are still open and thus may have content not yet flushed out. The fix is to close the files before redaction/zip.
| Status | Open [ 1 ] | In Progress [ 3 ] |
Build couchbase-server-7.0.1-5964 contains ns_server commit d25f38b with commit message:
MB-47593 Close files before doing post-processing (redact/zip)
| Resolution | Fixed [ 1 ] | |
| Status | In Progress [ 3 ] | Resolved [ 5 ] |
Build couchbase-server-7.1.0-1098 contains ns_server commit d25f38b with commit message:
MB-47593 Close files before doing post-processing (redact/zip)
| Fix Version/s | 7.0.2 [ 18012 ] |
| Fix Version/s | 7.0.1 [ 17104 ] |
| Labels | request-dev-verify |
| Status | Resolved [ 5 ] | Closed [ 6 ] |
On my mac running cheshire-cat sync'd as of this morning I see these files in the bundled .zip file (after unzipping):
$ ll *cprof*-rw-r--r-- 1 stevewatanabe staff 0 Jul 27 09:19 analytics_cprof.log-rw-r--r-- 1 stevewatanabe staff 0 Jul 27 09:19 eventing_cprof.log-rw-r--r-- 1 stevewatanabe staff 0 Jul 27 09:19 goxdcr_cprof.log-rw-r--r-- 1 stevewatanabe staff 0 Jul 27 09:18 indexer_cprof.log-rw-r--r-- 1 stevewatanabe staff 0 Jul 27 09:18 projector_cprof.log-rw-r--r-- 1 stevewatanabe staff 0 Jul 27 09:17 query_cprof.log...which I'm assuming means the cprof isn't available on mac. And...
$ ll master_events.log-rw-r--r-- 1 stevewatanabe staff 9717 Jul 27 09:20 master_events.log$ head -10 master_events.log{"type":"autofailoverNodeStateChange","ts":1627402555.861195,"prevState":"new","node":"n_0@cb.local","newState":"up","newCounter":0}{"type":"createBucket","ts":1627402573.567638,"bucketType":"membase","bucket":"default","params":{"storage_mode":"couchstore","replica_index":true,"ram_quota":1073741824,"num_threads":3,"num_replicas":1,"max_ttl":0,"flush_enabled":false,"eviction_policy":"value_only","durability_min_level":"none","conflict_resolution_type":"seqno","compression_mode":"passive"}}{"vbucket":0,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}{"vbucket":1,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}{"vbucket":2,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}{"vbucket":3,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}{"vbucket":4,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}{"vbucket":5,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}{"vbucket":6,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}{"vbucket":7,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}