Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49672

ARM: prometheus: panic signal SIGSEGV: segmentation violation

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes

    Description

      Build: 7.1.0-1738
      Operating System: Amazon Linux 2, distribution_version: Amazon Linux release 2 (Karoo) (ARM)

      http://server.jenkins.couchbase.com/job/build_sanity_matrix/9782/DISTRO=amzn2,TYPE=4node_2/consoleFull

      Seeing below panic in prometeous log in build sanity job run after installation with with 7.1.0-1738 with ARM.

      on 10.0.203.217:

      unexpected fault address 0x5f766b13627056
      fatal error: fault
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x5f766b13627056 pc=0x156f9b0]
       
      goroutine 307 [running]:
      runtime.throw(0x1d28f96, 0x5)
              /home/couchbase/jenkins/workspace/cbdeps-platform-build/deps/go1.16.5/src/runtime/panic.go:1117 +0x54 fp=0x40010dd530 sp=0x40010dd500 pc=0x44654
      runtime.sigpanic()
              /home/couchbase/jenkins/workspace/cbdeps-platform-build/deps/go1.16.5/src/runtime/signal_unix.go:741 +0x230 fp=0x40010dd570 sp=0x40010dd530 pc=0x5c130
      github.com/golang/snappy.encodeBlock(0x40010d4002, 0x5bd, 0x5bd, 0x4000254a00, 0x4d2, 0x500, 0x13)
              /home/couchbase/jenkins/workspace/cbdeps-platform-build/goproj/src/github.com/prometheus/prometheus/vendor/github.com/golang/snappy/encode_arm64.s:666 +0x360 fp=0x40010e5610 sp=0x40010dd580 pc=0x156f9b0
      github.com/golang/snappy.Encode(0x40010d4000, 0x5bf, 0x5bf, 0x0, 0x0, 0x0, 0x4d2, 0x1d4c2e3, 0x4000f9b758)
              /home/couchbase/jenkins/workspace/cbdeps-platform-build/goproj/src/github.com/prometheus/prometheus/vendor/github.com/golang/snappy/encode.go:39 +0x17c fp=0x40010e5660 sp=0x40010e5610 pc=0x156eeec
      github.com/prometheus/prometheus/tsdb/wal.(*WAL).log(0x40005c4510, 0x4000254a00, 0x4d2, 0x500, 0x1, 0xb, 0x4000756138)
              /home/couchbase/jenkins/workspace/cbdeps-platform-build/goproj/src/github.com/prometheus/prometheus/tsdb/wal/wal.go:634 +0x368 fp=0x40010e5700 sp=0x40010e5660 pc=0x15764c8
      github.com/prometheus/prometheus/tsdb/wal.(*WAL).Log(0x40005c4510, 0x40010e5808, 0x1, 0x1, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/cbdeps-platform-build/goproj/src/github.com/prometheus/prometheus/tsdb/wal/wal.go:596 +0xc8 fp=0x40010e5790 sp=0x40010e5700 pc=0x15760a8
      github.com/prometheus/prometheus/tsdb.(*headAppender).log(0x40010ab200, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/cbdeps-platform-build/goproj/src/github.com/prometheus/prometheus/tsdb/head.go:1212 +0x26c fp=0x40010e5840 sp=0x40010e5790 pc=0x1596f8c
      github.com/prometheus/prometheus/tsdb.(*headAppender).Commit(0x40010ab200, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/cbdeps-platform-build/goproj/src/github.com/prometheus/prometheus/tsdb/head.go:1232 +0x74 fp=0x40010e5980 sp=0x40010e5840 pc=0x15970e4
      github.com/prometheus/prometheus/tsdb.(*initAppender).Commit(0x40001254b8, 0x4000830780, 0x1d4627e)
              /home/couchbase/jenkins/workspace/cbdeps-platform-build/goproj/src/github.com/prometheus/prometheus/tsdb/head.go:1028 +0x38 fp=0x40010e59b0 sp=0x40010e5980 pc=0x1596058
      github.com/prometheus/prometheus/tsdb.dbAppender.Commit(0x2450148, 0x40001254b8, 0x40006fc000, 0x3, 0x256065fde24b4dba)
              /home/couchbase/jenkins/workspace/cbdeps-platform-build/goproj/src/github.com/prometheus/prometheus/tsdb/db.go:774 +0x30 fp=0x40010e59f0 sp=0x40010e59b0 pc=0x15888c0
      github.com/prometheus/prometheus/tsdb.(*dbAppender).Commit(0x40001254d0, 0x40007db8c0, 0x40007db8c0)
              <autogenerated>:1 +0x48 fp=0x40010e5a30 sp=0x40010e59f0 pc=0x15b21b8
      github.com/prometheus/prometheus/storage.(*fanoutAppe
      

      Logs:
      test_8.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          steve.watanabe Steve Watanabe added a comment - - edited

          Timofey Barmin From looking at https://prometheus.io/docs/guides/basic-auth/ it appears ns_server would have to use bcrypt to generate a hashed password which is then written to a file

          basic_auth_users:
              admin: $2b$12$hNf2lSsxfm0.i4a.1kVpSOVyBCfIB51VRjgBUyv6kdnyTlgWj81Ay
          

          and then the file name is passed to prometheus at start up

          prometheus -web.config.file=web.yml
          

          Assuming there's an acceptable bcrypt implementation in erlang this should be doable in neo.

          steve.watanabe Steve Watanabe added a comment - - edited Timofey Barmin From looking at https://prometheus.io/docs/guides/basic-auth/ it appears ns_server would have to use bcrypt to generate a hashed password which is then written to a file basic_auth_users: admin: $2b$12$hNf2lSsxfm0.i4a.1kVpSOVyBCfIB51VRjgBUyv6kdnyTlgWj81Ay and then the file name is passed to prometheus at start up prometheus -web.config.file=web.yml Assuming there's an acceptable bcrypt implementation in erlang this should be doable in neo.
          steve.watanabe Steve Watanabe added a comment - - edited

          I elected to update snappy to v0.0.3 to avoid the segfaults. I'll open a new ticket to track using the built-in prometheus basic auth. This is merged via https://github.com/couchbasedeps/prometheus/pull/16

          steve.watanabe Steve Watanabe added a comment - - edited I elected to update snappy to v0.0.3 to avoid the segfaults. I'll open a new ticket to track using the built-in prometheus basic auth. This is merged via https://github.com/couchbasedeps/prometheus/pull/16

          Build cbdeps::prometheus-2.23-2 contains prometheus commit f932c74 with commit message:
          MB-49672 Update prometheus to use snappy v0.0.3

          build-team Couchbase Build Team added a comment - Build cbdeps::prometheus-2.23-2 contains prometheus commit f932c74 with commit message: MB-49672 Update prometheus to use snappy v0.0.3

          Please retest with next build

          meni.hillel Meni Hillel (Inactive) added a comment - Please retest with next build

          Closing as we are not seeing this issue in the subsequent runs - 7.1.0-2235

          girish.benakappa Girish Benakappa added a comment - Closing as we are not seeing this issue in the subsequent runs - 7.1.0-2235

          People

            girish.benakappa Girish Benakappa
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty