Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40878

cbbackupmgr: confusing initial log message

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • Cheshire-Cat
    • 7.0.0
    • tools
    • None
    • Untriaged
    • 1
    • Unknown

    Description

      Problem
      The first message in the backup logs is a bit confusing:

      2020-08-11T13:32:54.558+00:00 (Cmd) cbbackupmgr version Unknown Hostname: ip-10-0-120-216.us-west-2.compute.internal OS: linux Version: unavailable Arch: amd64 vCPU: 4 Memory: unavailable
      

       Lots of "unknown" and "unavailable" in there.

      Steps to reproduce
      1. Create a repo using cbbackupmgr.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            james.lee James Lee added a comment -

            Hi Perry Krug,

            This is an interesting issue, looking at the log line we can see off the bat that you were hitting MB-41131 (which is why the version was reported as unknown). As for the Linux version/memory information being unavailable, it looks like we were unable to run the following commands (we know we will have attempted to run them because the OS version is correct):

            uname -mr
            cat /proc/meminfo
            

            This could be for many reasons but we don't log the errors so it's difficult to debug. Considering that we know this works on our own machines (and machines that I've run in GCE) I imagine this may be an environmental issue (for example permissions). Please could you provide some more information about the setup you were using at the time?

            james.lee James Lee added a comment - Hi Perry Krug , This is an interesting issue, looking at the log line we can see off the bat that you were hitting MB-41131 (which is why the version was reported as unknown). As for the Linux version/memory information being unavailable, it looks like we were unable to run the following commands (we know we will have attempted to run them because the OS version is correct): uname -mr cat /proc/meminfo This could be for many reasons but we don't log the errors so it's difficult to debug. Considering that we know this works on our own machines (and machines that I've run in GCE) I imagine this may be an environmental issue (for example permissions). Please could you provide some more information about the setup you were using at the time?
            perry Perry Krug added a comment -

            Thanks James. This is from a little while ago but I can see that it was an AWS instance in us-west-2, likely CentOS 7 or 8. I would have been root when running this so I doubt it is permissions based.

            I just tried it again with the latest 7.0 and it seems to be working fine now:

             

            [root@ip-10-0-236-167 logs]# head backup-0.log
            2020-11-30T18:15:15.363+00:00 (Cmd) cbbackupmgr version 7.0.0-3890 Hostname: ip-10-0-236-167.us-west-2.compute.internal OS: linux Version: 3.10.0-862.3.2.el7.x86_64 x86_64 Arch: amd64 vCPU: 4 Memory: 15791948

            {noforma}
            perry Perry Krug added a comment - Thanks James. This is from a little while ago but I can see that it was an AWS instance in us-west-2, likely CentOS 7 or 8. I would have been root when running this so I doubt it is permissions based. I just tried it again with the latest 7.0 and it seems to be working fine now:   [root@ip-10-0-236-167 logs] # head backup-0.log 2020-11-30T18:15:15.363+00:00 (Cmd) cbbackupmgr version 7.0.0-3890 Hostname: ip-10-0-236-167.us-west-2.compute.internal OS: linux Version: 3.10.0-862.3.2.el7.x86_64 x86_64 Arch: amd64 vCPU: 4 Memory: 15791948 {noforma}
            james.lee James Lee added a comment -

            Just an update on what I've been changing here. I've moved the system information utilities out into the tools-common library whilst updating them to support error handling. By that I mean we at return and log errors received whilst attempting to determine system information. This should mean that in the future any cases where we see 'unavailable', there should be an accompanying log message explaining why we're using the 'unavailable' placeholder.

            james.lee James Lee added a comment - Just an update on what I've been changing here. I've moved the system information utilities out into the tools-common library whilst updating them to support error handling. By that I mean we at return and log errors received whilst attempting to determine system information. This should mean that in the future any cases where we see 'unavailable', there should be an accompanying log message explaining why we're using the 'unavailable' placeholder.

            Build couchbase-server-7.0.0-4633 contains backup commit 47b8eba with commit message:
            MB-40878 Switch over to system info utilities in tools-common

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4633 contains backup commit 47b8eba with commit message: MB-40878 Switch over to system info utilities in tools-common

            Build couchbase-server-7.0.0-4639 contains backup commit e1d6749 with commit message:
            MB-40878 Update the tools-common dependency

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4639 contains backup commit e1d6749 with commit message: MB-40878 Update the tools-common dependency
            james.lee James Lee added a comment -

            I'm going to mark this as resolved for the time being, I've updated the way that we determine the system memory/version so that it now correctly handles errors, and no longer conditionally runs commands at runtime i.e. the right commands are determined at compile time.

            If we see unknown/unavailable in the logs, we'll not see an accompanying error indicating why we failed to determine the system version/memory.

            james.lee James Lee added a comment - I'm going to mark this as resolved for the time being, I've updated the way that we determine the system memory/version so that it now correctly handles errors, and no longer conditionally runs commands at runtime i.e. the right commands are determined at compile time. If we see unknown/unavailable in the logs, we'll not see an accompanying error indicating why we failed to determine the system version/memory.
            james.lee James Lee added a comment -

            Hi Carlos Gonzalez Betancort,

            I've reopened this an assigned it to you as this appears to be an issue with cbbs; more specifically cbbs does not appear to be passing the existing environment down to cbm (this may be desired, however, we should be at least passing select variables e.g. PATH).

            I've reproduced this issue using a test server from QE and we can clearly see this issue now that we're no longer ignoring the error message.

            uname not in $PATH

            2021-03-15T13:30:13.611-07:00 ERRO: failed to get system version: failed to run 'uname -r': exec: "uname": executable file not found in $PATH -- logging.(*ToolsCommonLogger).Log() at tools_common.go:30
            

             
            Please note that uname does exist on this machine, and running without using cbbs results in the version being detected correctly.

            Looking at the second example for exec.Command is appears that we should be passing down os.Environ() to cbbackupmgr. I understand that this might not be possible/viable due to cbauth environment variables, however, we should at least be passing the PATH environment variable.

            Please could you look at getting a patch up for this.

            james.lee James Lee added a comment - Hi Carlos Gonzalez Betancort , I've reopened this an assigned it to you as this appears to be an issue with cbbs; more specifically cbbs does not appear to be passing the existing environment down to cbm (this may be desired, however, we should be at least passing select variables e.g. PATH). I've reproduced this issue using a test server from QE and we can clearly see this issue now that we're no longer ignoring the error message. uname not in $PATH 2021-03-15T13:30:13.611-07:00 ERRO: failed to get system version: failed to run 'uname -r': exec: "uname": executable file not found in $PATH -- logging.(*ToolsCommonLogger).Log() at tools_common.go:30   Please note that uname does exist on this machine, and running without using cbbs results in the version being detected correctly. Looking at the second example for exec.Command is appears that we should be passing down os.Environ() to cbbackupmgr. I understand that this might not be possible/viable due to cbauth environment variables, however, we should at least be passing the PATH environment variable. Please could you look at getting a patch up for this.

            James Lee I'll try and get a paatch Up fist, i am going to check that ns_server gives us the path, if not I will have to pass it along.

            carlos.gonzalez Carlos Gonzalez Betancort (Inactive) added a comment - James Lee I'll try and get a paatch Up fist, i am going to check that ns_server gives us the path, if not I will have to pass it along.

            Build couchbase-server-7.0.0-4704 contains cbbs commit 220ae80 with commit message:
            MB-40878 Pass the PATH env var to CBM

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4704 contains cbbs commit 220ae80 with commit message: MB-40878 Pass the PATH env var to CBM

            Can reproduce the issue by simply deleting the uname binary, looks like cbbackupmgr logs why it failed to get the system version.

            2021-03-18T10:19:57.241+00:00 ERRO: failed to get system version: failed to run 'uname -r':  -- logging.(*ToolsCommonLogger).Log() at tools_common.go:30
            2021-03-18T10:19:57.241+00:00 (Cmd) cbbackupmgr version 7.0.0-4678 Hostname: node1-cheshire-cat-testing-centos7.vagrants OS: linux Version: unavailable Arch: amd64 vCPU: 1 Memory: 2912301056 (2.71GiB)
            

            Closing.

            asad.zaidi Asad Zaidi (Inactive) added a comment - Can reproduce the issue by simply deleting the uname binary, looks like cbbackupmgr logs why it failed to get the system version. 2021-03-18T10:19:57.241+00:00 ERRO: failed to get system version: failed to run 'uname -r': -- logging.(*ToolsCommonLogger).Log() at tools_common.go:30 2021-03-18T10:19:57.241+00:00 (Cmd) cbbackupmgr version 7.0.0-4678 Hostname: node1-cheshire-cat-testing-centos7.vagrants OS: linux Version: unavailable Arch: amd64 vCPU: 1 Memory: 2912301056 (2.71GiB) Closing.

            People

              carlos.gonzalez Carlos Gonzalez Betancort (Inactive)
              perry Perry Krug
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty