Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50153

Node goes down after attempting to delete security/responseHeaders

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • Unknown

    Description

      Steps to Repro
      1. Create a 1 node KV cluster with node: 172.23.107.90 = sa1711.sc.couchbase.com
      2. Execute

      curl -v -u Administrator:password http://sa1711.sc.couchbase.com:8091/internalSettings -d "canEnableStrictEncryption=true" 

      3. Set response headers for HSTS

      curl -u Administrator:password -H "Content-Type: application/json" -X POST http://sa1711.sc.couchbase.com:8091/settings/security/responseHeaders -d '{"Strict-Transport-Security": "max-age=300;includeSubDomains;preload"}' 

      4. Check if audit is generated. 

      {"description":"Security Settings","id":8237,"name":"security settings","real_userid":{"domain":"builtin","user":"Administrator"},"remote":{"ip":"172.23.107.90","port":60462},"settings":{"secure_headers":{"Strict-Transport-Security":"max-age=300;includeSubDomains;preload"}},"timestamp":"2021-12-17T09:51:07.613-08:00"}

      5. Attempt to delete headers

      curl -v -u Administrator:password -X DELETE https://172.23.107.90:18091/settings/security/responseHeaders -k

      we receive empty response from the server. And the cluster becomes unreachable. But audit is generated

      {"description":"Security Settings","id":8237,"name":"security settings","real_userid":{"domain":"builtin","user":"Administrator"},"remote":{"ip":"172.23.107.90","port":33510},"settings":{"secure_headers":"deleted"},"timestamp":"2021-12-17T09:52:35.152-08:00"}

      Doing the same on 7.0.3 for node 172.23.106.237 = sa1712.sc.couchbase.com works fine. Cluster is healthy in 7.0.3.

      Observations from 6.6.5 
      In error.log of .90

      [ns_server:error,2021-12-17T09:52:35.153-08:00,ns_1@cb.local:<0.593.0>:menelaus_web:loop:171]Server error during processing: ["web request failed",
                                       {path,"/settings/security/responseHeaders"},
                                       {method,'DELETE'},
                                       {type,error},
                                       {what,{badmatch,false}},
                                       {trace,
                                        [{menelaus_util,compute_sec_headers,0,
                                          [{file,"src/menelaus_util.erl"},
                                           {line,96}]},
                                         {menelaus_util,response_headers,1,
                                          [{file,"src/menelaus_util.erl"},
                                           {line,139}]},
                                         {menelaus_util,reply_ok,4,
                                          [{file,"src/menelaus_util.erl"},
                                           {line,209}]},
                                         {request_throttler,do_request,3,
                                          [{file,"src/request_throttler.erl"},
                                           {line,59}]},
                                         {menelaus_web,loop,2,
                                          [{file,"src/menelaus_web.erl"},
                                           {line,149}]},
                                         {mochiweb_http,headers,5,
                                          [{file,
                                            "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/mochiweb/mochiweb_http.erl"},
                                           {line,94}]},
                                         {proc_lib,init_p_do_apply,3,
                                          [{file,"proc_lib.erl"},{line,247}]}]}]
      [ns_server:error,2021-12-17T09:52:35.836-08:00,ns_1@cb.local:wait_link_to_couchdb_node<0.8780.0>:ns_server_nodes_sup:do_wait_link_to_couchdb_node:192]ns_couchdb_port(<0.279.0>) died with reason {abnormal,1}

      in debug.log

      =========================INFO REPORT=========================
      {net_kernel,{'EXIT',<0.25975.1>,{recv_challenge_ack_failed,{error,closed}}}}
      [ns_server:debug,2021-12-17T10:10:27.266-08:00,ns_1@cb.local:<0.25964.1>:ns_server_nodes_sup:do_wait_link_to_couchdb_node:169]ns_couchdb is not ready: {badrpc,nodedown}
      [ns_server:debug,2021-12-17T10:10:27.266-08:00,ns_1@cb.local:cb_dist<0.175.0>:cb_dist:info_msg:809]cb_dist: Connection down: {con,#Ref<0.1676609119.2549874689.26010>,
                                     inet_tcp_dist,<0.25975.1>,
                                     #Ref<0.1676609119.2549874689.26012>}
      [error_logger:info,2021-12-17T10:10:27.267-08:00,ns_1@cb.local:error_logger<0.32.0>:ale_error_logger_handler:do_log:203]
      =========================INFO REPORT=========================

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Build couchbase-server-7.1.0-1955 contains ns_server commit abce0aa with commit message:
          MB-50153: Merge remote-tracking branch 'couchbase/mad-hatter'

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1955 contains ns_server commit abce0aa with commit message: MB-50153 : Merge remote-tracking branch 'couchbase/mad-hatter'

          Build couchbase-server-7.1.0-1955 contains ns_server commit 5084c45 with commit message:
          MB-50153: Fix crash when secure_headers is missing

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1955 contains ns_server commit 5084c45 with commit message: MB-50153 : Fix crash when secure_headers is missing

          Build couchbase-server-6.6.5-10066 contains ns_server commit 5084c45 with commit message:
          MB-50153: Fix crash when secure_headers is missing

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.5-10066 contains ns_server commit 5084c45 with commit message: MB-50153 : Fix crash when secure_headers is missing

          Build couchbase-server-7.0.4-7206 contains ns_server commit abce0aa with commit message:
          MB-50153: Merge remote-tracking branch 'couchbase/mad-hatter'

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.4-7206 contains ns_server commit abce0aa with commit message: MB-50153 : Merge remote-tracking branch 'couchbase/mad-hatter'

          Build couchbase-server-7.0.4-7206 contains ns_server commit 5084c45 with commit message:
          MB-50153: Fix crash when secure_headers is missing

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.4-7206 contains ns_server commit 5084c45 with commit message: MB-50153 : Fix crash when secure_headers is missing

          Verified on 6.6.5-10070. Closing. 

          sumedh.basarkod Sumedh Basarkod (Inactive) added a comment - Verified on 6.6.5-10070. Closing. 

          People

            sumedh.basarkod Sumedh Basarkod (Inactive)
            sumedh.basarkod Sumedh Basarkod (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty