Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37302

windows - n2n - fails with http500

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Test Blocker
    • Resolution: Fixed
    • 6.5.0
    • 6.5.0
    • build, ns_server
    • Enterprise Edition 6.5.0 build 4947 ‧ IPv4 ‧ encrypted

    Description

      1. On windows box, enable n2n encryption while node init.
      2. Add in another windows node

      Add Node fails:

      Failed to add node 172.23.121.187:18091 to cluster. Join completion call failed. Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: "[\"Unexpected server error, request logged.\"]"
      ns_cluster 005
      ns_1@172.23.121.167
      2:54:27 AM   18 Dec, 2019
      Add transaction of 'ns_1@172.23.121.187' failed because of {error,
      complete_join,
      <<"Join completion call failed. Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\"">>,
      {error,
      rest_error,
      <<"Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\"">>,
      {bad_status,500,
      <<"[\"Unexpected server error, request logged.\"]">>}}}
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          ritam.sharma Ritam Sharma created issue -
          ritam.sharma Ritam Sharma made changes -
          Field Original Value New Value
          Link to Log File, atop/blg, CBCollectInfo, Core dump https://s3.amazonaws.com/cb-engineering/jira/MB-37302-node1/collectinfo-2019-12-18T105942-ns_1%40172.23.121.167.zip
          ritam.sharma Ritam Sharma added a comment -

          Path is incorrect for key file -
          Cannot open TLS distribution connection: Invalid key file C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem: no such file or directory

          [ns_server:debug,2019-12-18T03:08:39.897-08:00,ns_1@172.23.121.187:cb_dist<0.20873.1>:cb_dist:info_msg:754]cb_dist: Connection down: {con,#Ref<0.123361788.2236612614.125175>,
          inet_tls_dist,<0.11584.2>,

          ritam.sharma Ritam Sharma added a comment - Path is incorrect for key file - Cannot open TLS distribution connection: Invalid key file C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem: no such file or directory [ns_server:debug,2019-12-18T03:08:39.897-08:00,ns_1@172.23.121.187:cb_dist<0.20873.1>:cb_dist:info_msg:754] cb_dist: Connection down: {con,#Ref<0.123361788.2236612614.125175>, inet_tls_dist,<0.11584.2>,
          dfinlay Dave Finlay made changes -
          Assignee Dave Finlay [ dfinlay ] Timofey Barmin [ timofey.barmin ]
          dfinlay Dave Finlay added a comment -

          Timofey Barmin: can you investigate this issue?

          dfinlay Dave Finlay added a comment - Timofey Barmin : can you investigate this issue?
          ritam.sharma Ritam Sharma made changes -
          Description 1. On windows box, enable n2n encryption while node init.
          2. Rebalance in another windows node

          Rebalance fails:
          {code}
          Failed to add node 172.23.121.187:18091 to cluster. Join completion call failed. Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: "[\"Unexpected server error, request logged.\"]"
          ns_cluster 005
          ns_1@172.23.121.167
          2:54:27 AM 18 Dec, 2019
          Add transaction of 'ns_1@172.23.121.187' failed because of {error,
          complete_join,
          <<"Join completion call failed. Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\"">>,
          {error,
          rest_error,
          <<"Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\"">>,
          {bad_status,500,
          <<"[\"Unexpected server error, request logged.\"]">>}}}
          {code}
          1. On windows box, enable n2n encryption while node init.
          2. Add in another windows node

          Rebalance fails:
          {code}
          Failed to add node 172.23.121.187:18091 to cluster. Join completion call failed. Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: "[\"Unexpected server error, request logged.\"]"
          ns_cluster 005
          ns_1@172.23.121.167
          2:54:27 AM 18 Dec, 2019
          Add transaction of 'ns_1@172.23.121.187' failed because of {error,
          complete_join,
          <<"Join completion call failed. Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\"">>,
          {error,
          rest_error,
          <<"Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\"">>,
          {bad_status,500,
          <<"[\"Unexpected server error, request logged.\"]">>}}}
          {code}
          ritam.sharma Ritam Sharma made changes -
          Description 1. On windows box, enable n2n encryption while node init.
          2. Add in another windows node

          Rebalance fails:
          {code}
          Failed to add node 172.23.121.187:18091 to cluster. Join completion call failed. Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: "[\"Unexpected server error, request logged.\"]"
          ns_cluster 005
          ns_1@172.23.121.167
          2:54:27 AM 18 Dec, 2019
          Add transaction of 'ns_1@172.23.121.187' failed because of {error,
          complete_join,
          <<"Join completion call failed. Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\"">>,
          {error,
          rest_error,
          <<"Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\"">>,
          {bad_status,500,
          <<"[\"Unexpected server error, request logged.\"]">>}}}
          {code}
          1. On windows box, enable n2n encryption while node init.
          2. Add in another windows node

          Add Node fails:
          {code}
          Failed to add node 172.23.121.187:18091 to cluster. Join completion call failed. Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: "[\"Unexpected server error, request logged.\"]"
          ns_cluster 005
          ns_1@172.23.121.167
          2:54:27 AM 18 Dec, 2019
          Add transaction of 'ns_1@172.23.121.187' failed because of {error,
          complete_join,
          <<"Join completion call failed. Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\"">>,
          {error,
          rest_error,
          <<"Got HTTP status 500 from REST call post to https://172.23.121.187:18091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\"">>,
          {bad_status,500,
          <<"[\"Unexpected server error, request logged.\"]">>}}}
          {code}

          Confirm. Looks like the problem is missing key file:

          [error_logger:error,2019-12-18T02:54:24.282-08:00,ns_1@172.23.121.187:error_logger<0.32.0>:ale_error_logger_handler:do_log:203]Cannot open TLS distribution connection: Invalid key file C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem: no such file or directory
          

          It's not clear why it's missing though. Still investigating.

          timofey.barmin Timofey Barmin added a comment - Confirm. Looks like the problem is missing key file: [error_logger:error,2019-12-18T02:54:24.282-08:00,ns_1@172.23.121.187:error_logger<0.32.0>:ale_error_logger_handler:do_log:203]Cannot open TLS distribution connection: Invalid key file C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem: no such file or directory It's not clear why it's missing though. Still investigating.

          Unfortunately couchbase.log doesn't have "Directory structure" on Windows, so I can't be sure but it looks like we store key file in one directory and try to read it from other directory.
          For example web server uses key file located in c:/Program Files/Couchbase/Server/var/lib/couchbase/config

          =====================PROGRESS REPORT=========================
                 supervisor: {<0.20796.1>,menelaus_web}
                    started: [{pid,<0.20815.1>},
                              {id,menelaus_web_ipv6},
                              {mfargs,
                               {menelaus_web,http_server,
                                [[{ip,"::"},
                                  {name,menelaus_web_ssl_ipv6},
                                  {ssl,true},
                                  {ssl_opts,
                                   [{keyfile,
                                     "c:/Program Files/Couchbase/Server/var/lib/couchbase/config/ssl-cert-key.pem"},
          

          but netkernel is looking for key file in C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/

          timofey.barmin Timofey Barmin added a comment - Unfortunately couchbase.log doesn't have "Directory structure" on Windows, so I can't be sure but it looks like we store key file in one directory and try to read it from other directory. For example web server uses key file located in c:/Program Files/Couchbase/Server/var/lib/couchbase/config =====================PROGRESS REPORT========================= supervisor: {<0.20796.1>,menelaus_web} started: [{pid,<0.20815.1>}, {id,menelaus_web_ipv6}, {mfargs, {menelaus_web,http_server, [[{ip,"::"}, {name,menelaus_web_ssl_ipv6}, {ssl,true}, {ssl_opts, [{keyfile, "c:/Program Files/Couchbase/Server/var/lib/couchbase/config/ssl-cert-key.pem"}, but netkernel is looking for key file in C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/

          Contents of ssl_dist_opts file can be taken from ets table:

           38774 per_node_ets_tables('ns_1@172.23.121.187', ssl_dist_opts) =
           38775   Info: [{read_concurrency,false},
           38776          {write_concurrency,false},
           38777          {compressed,false},
           38778          {memory,1520},
           38779          {owner,<0.20864.1>},
           38780          {heir,none},
           38781          {name,ssl_dist_opts},
           38782          {size,2},
           38783          {node,'ns_1@172.23.121.187'},
           38784          {named_table,true},
           38785          {type,set},
           38786          {keypos,1},
           38787          {protection,public}]
           38788   Values:-
           38789     {server,[{fail_if_no_peer_cert,true},
           38790              {certfile,"C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem"},
           38791              {keyfile,"C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem"},
           38792              {cacertfile,"C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem-ca"},
           38793              {reuse_sessions,false}]}
           38794     {client,[{certfile,"C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem"},
           38795              {keyfile,"C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem"},
           38796              {cacertfile,"C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem-ca"},
           38797              {verify,verify_peer},
           38798              {reuse_sessions,false}]}
          

          timofey.barmin Timofey Barmin added a comment - Contents of ssl_dist_opts file can be taken from ets table: 38774 per_node_ets_tables('ns_1@172.23.121.187', ssl_dist_opts) = 38775 Info: [{read_concurrency,false}, 38776 {write_concurrency,false}, 38777 {compressed,false}, 38778 {memory,1520}, 38779 {owner,<0.20864.1>}, 38780 {heir,none}, 38781 {name,ssl_dist_opts}, 38782 {size,2}, 38783 {node,'ns_1@172.23.121.187'}, 38784 {named_table,true}, 38785 {type,set}, 38786 {keypos,1}, 38787 {protection,public}] 38788 Values:- 38789 {server,[{fail_if_no_peer_cert,true}, 38790 {certfile,"C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem"}, 38791 {keyfile,"C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem"}, 38792 {cacertfile,"C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem-ca"}, 38793 {reuse_sessions,false}]} 38794 {client,[{certfile,"C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem"}, 38795 {keyfile,"C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem"}, 38796 {cacertfile,"C:/Jenkins/workspace/couchbase-server-windows/install/var/lib/couchbase/config/ssl-cert-key.pem-ca"}, 38797 {verify,verify_peer}, 38798 {reuse_sessions,false}]}
          dfinlay Dave Finlay made changes -
          Link This issue blocks MB-36676 [ MB-36676 ]
          dfinlay Dave Finlay made changes -
          Labels approved-for-mad-hatter
          dfinlay Dave Finlay made changes -
          Component/s build [ 10200 ]

          Build couchbase-server-6.5.0-4956 contains voltron commit 4518848 with commit message:
          MB-37302: Replace paths in ssl_dist_opts file with real ...

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4956 contains voltron commit 4518848 with commit message: MB-37302 : Replace paths in ssl_dist_opts file with real ...

          Build couchbase-server-6.5.1-6020 contains voltron commit 4518848 with commit message:
          MB-37302: Replace paths in ssl_dist_opts file with real ...

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.1-6020 contains voltron commit 4518848 with commit message: MB-37302 : Replace paths in ssl_dist_opts file with real ...
          dfinlay Dave Finlay made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          ritam.sharma Ritam Sharma added a comment - - edited

          Dave Finlay and Timofey Barmin - Initial test looks good. Will run more test. Thank You.
          Checked with non root install as well, since install path is different that regular one; no issues there as well.
          ipv6 on windows also pass and regression is looking good too.

          ritam.sharma Ritam Sharma added a comment - - edited Dave Finlay and Timofey Barmin - Initial test looks good. Will run more test. Thank You. Checked with non root install as well, since install path is different that regular one; no issues there as well. ipv6 on windows also pass and regression is looking good too.

          Great! Thanks Ritam Sharma

          timofey.barmin Timofey Barmin added a comment - Great! Thanks Ritam Sharma

          Build couchbase-server-7.0.0-1145 contains voltron commit 4518848 with commit message:
          MB-37302: Replace paths in ssl_dist_opts file with real ...

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1145 contains voltron commit 4518848 with commit message: MB-37302 : Replace paths in ssl_dist_opts file with real ...
          ritam.sharma Ritam Sharma made changes -
          VERIFICATION STEPS 6.5.0-4956 - closed based on comments by Ritam.
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            timofey.barmin Timofey Barmin
            ritam.sharma Ritam Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty