Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47593

cbcollect_info - few files are 0 bytes in cbcollect zip

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes

    Description

      I found some oddity that shows up when trying to create a backport for MB-47380 into 7.0.1.
      MB-47380 has been fine and it still works fine in master.

      However, once cherry-picked the portions to CC, re-running cbcollect (without redaction) ends up with empty files that MB-47380 added... namely, goxdcr_remote_clusters.json and goxdcr_replications.json.

      Spoke with Steve Watanabe, and he pointed me to a few check-ins lately that also touched cbcollect_info. Namely:
      http://review.couchbase.org/c/ns_server/+/157902
      and
      http://review.couchbase.org/c/ns_server/+/157903

      Interestingly, when I reverted http://review.couchbase.org/c/ns_server/+/157902 patch locally, the cbcollect works, which means that the newly added XDCR files have content in them.

      It also seems like the same situation of 0 byte files in the zip file also occurs to at least one other file. One other example I found used a similar syntax as the new goxdcr files being generated. Specifically, the entry:

              make_curl_task(name="master events",
                             user="@",
                             password=memcached_pass,
                             timeout=300,
                             url='http://%s:%s/diag/masterEvents?o=1' % (
                                 local_url_addr, read_guts(guts, "rest_port")),
                             log_file="master_events.log",
                             no_header=True),
      

      (And another non-text file, projector_cprof.log)

      Ultimately, without reverting the patch, the 2 goxdcr files + the master_events.log + projector_cprof.log are empty in the zip file. And, reverting the patch seems to flush them just fine (non-0 bytes and with valid data). Thus, I don't think it is specific only to goxdcr portion of the log collection.

      Visually, I think the patch looks fine, and atm I don't see anything wrong with it.

      What's weird is also the fact that all 3 patches (the 2 mentioned above + goxdcr's portion) are present in master, and cbcollect_info runs just fine in master.
      File size was a thought, but there are other small files that were flushed just fine with the changeset.
      File descriptor count doesn't seem to be a problem, as ulimit -n is big enough on my machine.

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-47593
          # Subject Branch Project Status CR V

          Activity

            Build couchbase-server-7.1.0-1098 contains ns_server commit d25f38b with commit message:
            MB-47593 Close files before doing post-processing (redact/zip)

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1098 contains ns_server commit d25f38b with commit message: MB-47593 Close files before doing post-processing (redact/zip)

            Build couchbase-server-7.0.1-5964 contains ns_server commit d25f38b with commit message:
            MB-47593 Close files before doing post-processing (redact/zip)

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.1-5964 contains ns_server commit d25f38b with commit message: MB-47593 Close files before doing post-processing (redact/zip)

            The problem is that we redact/zip files that are still open and thus may have content not yet flushed out. The fix is to close the files before redaction/zip.

            steve.watanabe Steve Watanabe added a comment - The problem is that we redact/zip files that are still open and thus may have content not yet flushed out. The fix is to close the files before redaction/zip.

            Without the two recent cbcollect_info changes there is content in the cprof file

            $ ll *cprof*
            -rw-r--r--  1 stevewatanabe  staff    16 Jul 27 09:38 analytics_cprof.log
            -rw-r--r--  1 stevewatanabe  staff   543 Jul 27 09:38 eventing_cprof.log
            -rw-r--r--  1 stevewatanabe  staff  1323 Jul 27 09:39 goxdcr_cprof.log
            -rw-r--r--  1 stevewatanabe  staff  2830 Jul 27 09:37 indexer_cprof.log
            -rw-r--r--  1 stevewatanabe  staff   107 Jul 27 09:38 projector_cprof.log
            -rw-r--r--  1 stevewatanabe  staff  1604 Jul 27 09:37 query_cprof.log 

            seems to be related to log files with no_header=True

            steve.watanabe Steve Watanabe added a comment - Without the two recent cbcollect_info changes there is content in the cprof file $ ll *cprof* -rw-r--r-- 1 stevewatanabe staff 16 Jul 27 09:38 analytics_cprof.log -rw-r--r-- 1 stevewatanabe staff 543 Jul 27 09:38 eventing_cprof.log -rw-r--r-- 1 stevewatanabe staff 1323 Jul 27 09:39 goxdcr_cprof.log -rw-r--r-- 1 stevewatanabe staff 2830 Jul 27 09:37 indexer_cprof.log -rw-r--r-- 1 stevewatanabe staff 107 Jul 27 09:38 projector_cprof.log -rw-r--r-- 1 stevewatanabe staff 1604 Jul 27 09:37 query_cprof.log seems to be related to log files with no_header=True

            On my mac running cheshire-cat sync'd as of this morning I see these files in the bundled .zip file (after unzipping):

            $ ll *cprof*
            -rw-r--r--  1 stevewatanabe  staff  0 Jul 27 09:19 analytics_cprof.log
            -rw-r--r--  1 stevewatanabe  staff  0 Jul 27 09:19 eventing_cprof.log
            -rw-r--r--  1 stevewatanabe  staff  0 Jul 27 09:19 goxdcr_cprof.log
            -rw-r--r--  1 stevewatanabe  staff  0 Jul 27 09:18 indexer_cprof.log
            -rw-r--r--  1 stevewatanabe  staff  0 Jul 27 09:18 projector_cprof.log
            -rw-r--r--  1 stevewatanabe  staff  0 Jul 27 09:17 query_cprof.log 

            ...which I'm assuming means the cprof isn't available on mac. And...

            $ ll master_events.log
            -rw-r--r--  1 stevewatanabe  staff  9717 Jul 27 09:20 master_events.log
             
            $ head -10 master_events.log
            {"type":"autofailoverNodeStateChange","ts":1627402555.861195,"prevState":"new","node":"n_0@cb.local","newState":"up","newCounter":0}
            {"type":"createBucket","ts":1627402573.567638,"bucketType":"membase","bucket":"default","params":{"storage_mode":"couchstore","replica_index":true,"ram_quota":1073741824,"num_threads":3,"num_replicas":1,"max_ttl":0,"flush_enabled":false,"eviction_policy":"value_only","durability_min_level":"none","conflict_resolution_type":"seqno","compression_mode":"passive"}}
            {"vbucket":0,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}
            {"vbucket":1,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}
            {"vbucket":2,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}
            {"vbucket":3,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}
            {"vbucket":4,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}
            {"vbucket":5,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}
            {"vbucket":6,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}
            {"vbucket":7,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]} 

             

            steve.watanabe Steve Watanabe added a comment - On my mac running cheshire-cat sync'd as of this morning I see these files in the bundled .zip file (after unzipping): $ ll *cprof* -rw-r--r-- 1 stevewatanabe staff 0 Jul 27 09:19 analytics_cprof.log -rw-r--r-- 1 stevewatanabe staff 0 Jul 27 09:19 eventing_cprof.log -rw-r--r-- 1 stevewatanabe staff 0 Jul 27 09:19 goxdcr_cprof.log -rw-r--r-- 1 stevewatanabe staff 0 Jul 27 09:18 indexer_cprof.log -rw-r--r-- 1 stevewatanabe staff 0 Jul 27 09:18 projector_cprof.log -rw-r--r-- 1 stevewatanabe staff 0 Jul 27 09:17 query_cprof.log ...which I'm assuming means the cprof isn't available on mac. And... $ ll master_events.log -rw-r--r-- 1 stevewatanabe staff 9717 Jul 27 09:20 master_events.log   $ head -10 master_events.log {"type":"autofailoverNodeStateChange","ts":1627402555.861195,"prevState":"new","node":"n_0@cb.local","newState":"up","newCounter":0} {"type":"createBucket","ts":1627402573.567638,"bucketType":"membase","bucket":"default","params":{"storage_mode":"couchstore","replica_index":true,"ram_quota":1073741824,"num_threads":3,"num_replicas":1,"max_ttl":0,"flush_enabled":false,"eviction_policy":"value_only","durability_min_level":"none","conflict_resolution_type":"seqno","compression_mode":"passive"}} {"vbucket":0,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]} {"vbucket":1,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]} {"vbucket":2,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]} {"vbucket":3,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]} {"vbucket":4,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]} {"vbucket":5,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]} {"vbucket":6,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]} {"vbucket":7,"type":"updateMap","ts":1627402573.813229,"bucket":"default","chainBefore":[],"chainAfter":["127.0.0.1:11999",""]}  

            People

              steve.watanabe Steve Watanabe
              neil.huang Neil Huang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty