Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51197

cbcollect - failes to gather projector profiles when TLS is enforced

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      cbcollect collects the projector CPU, memory and goroutine dumps of projector process. This is a critical piece of information which will help us debug customer CBSE's.

      So far, this information is gathered using REST requests on 127.0.0.1:9999 port using HTTP. After TLS is enabled, projector uses the same port for TLS communication as well. So, any HTTP REST request would return: "Client sent an HTTP request to an HTTPS server". Enabling HTTPS request in cbcollect requires cacert.

       

      Ask of this MB:

      a. Does cbcollect know if TLS is enforced or not so that it can query with the appropriate protocol?

      b. If cbcollect is unaware of it TLS, does each service need to query both protocols?

      c. Is cacert available to cbcollect so that it can query on HTTPS protocol?

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            varun.velamuri Varun Velamuri created issue -
            amit.kulkarni Amit Kulkarni made changes -
            Field Original Value New Value
            Labels backport-candidate candidate-for-7.0.3 backport-candidate candidate-for-7.0.3 candidate-for-7.0.4
            dfinlay Dave Finlay made changes -
            Assignee Dave Finlay [ dfinlay ] Bryan McCoid [ JIRAUSER25453 ]
            bryan.mccoid Bryan McCoid made changes -
            Assignee Bryan McCoid [ JIRAUSER25453 ] Varun Velamuri [ varun.velamuri ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]

            Build couchbase-server-7.1.0-2413 contains ns_server commit 9d84daa with commit message:
            MB-51197: Add tls status of node to dump-guts/cbcollect

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2413 contains ns_server commit 9d84daa with commit message: MB-51197 : Add tls status of node to dump-guts/cbcollect
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Assignee Varun Velamuri [ varun.velamuri ] Hemant Rajput [ hemant.rajput ]
            hemant.rajput Hemant Rajput added a comment - - edited

             

            Projector's cpu, memory and go dump are not accessible when TLS is enabled on build 7.1.0-2413

             

             

            lfc@LFCs-MacBook-Pro cbcollect_info_ns_1@10.112.205.101_20220302-065756 % cat projector_cprof.log 
            curl: (60) SSL certificate problem: unable to get local issuer certificate
            More details here: https://curl.se/docs/sslcerts.html
             
             
            curl failed to verify the legitimacy of the server and therefore could not
            establish a secure connection to it. To learn more about this situation and
            how to fix it, please visit the web page mentioned above.
            lfc@LFCs-MacBook-Pro cbcollect_info_ns_1@10.112.205.101_20220302-065756 % cat projector_mprof.log 
            curl: (60) SSL certificate problem: unable to get local issuer certificate
            More details here: https://curl.se/docs/sslcerts.html
             
             
            curl failed to verify the legitimacy of the server and therefore could not
            establish a secure connection to it. To learn more about this situation and
            how to fix it, please visit the web page mentioned above.
            lfc@LFCs-MacBook-Pro cbcollect_info_ns_1@10.112.205.101_20220302-065756 % cat projector_pprof.log
            curl: (60) SSL certificate problem: unable to get local issuer certificate
            More details here: https://curl.se/docs/sslcerts.html
             
             
            curl failed to verify the legitimacy of the server and therefore could not
            establish a secure connection to it. To learn more about this situation and
            how to fix it, please visit the web page mentioned above.
            lfc@LFCs-MacBook-Pro cbcollect_info_ns_1@10.112.205.101_20220302-065756 %

             

             

            Steps to reproduce:

            1. Create a 2 node cluster with following services kv:n1ql-index.
            2. Set index storage mode
            3. Enable TLS

              ./couchbase-cli setting-autofailover -c localhost:8091 -u Administrator -p password --enable-auto-failover=0
              ./couchbase-cli node-to-node-encryption -c localhost:8091 -u Administrator -p password --enable
              ./couchbase-cli setting-security -c localhost:8091 -u Administrator -p password --set --cluster-encryption-level all
              ./couchbase-cli setting-autofailover -c localhost:8091 -u Administrator -p password --enable-auto-failover=1 --auto-failover-timeout=120 --max-failovers=1
              ./couchbase-cli setting-security -c https://localhost:18091 -u Administrator -p password --set --cluster-encryption-level strict --no-ssl-verify 

            4. Load travel-sample bucket.
            5. Run cbcollect_info on both the nodes - /opt/couchbase/bin/cbcollect_info  /tmp/$(hostname).zip

            I've added the cbcollect zips for reference.

            hemant.rajput Hemant Rajput added a comment - - edited   Projector's cpu, memory and go dump are not accessible when TLS is enabled on build 7.1.0-2413     lfc@LFCs-MacBook-Pro cbcollect_info_ns_1@10.112.205.101_20220302-065756 % cat projector_cprof.log  curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.se/docs/sslcerts.html     curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. lfc@LFCs-MacBook-Pro cbcollect_info_ns_1@10.112.205.101_20220302-065756 % cat projector_mprof.log  curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.se/docs/sslcerts.html     curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. lfc@LFCs-MacBook-Pro cbcollect_info_ns_1@10.112.205.101_20220302-065756 % cat projector_pprof.log curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.se/docs/sslcerts.html     curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. lfc@LFCs-MacBook-Pro cbcollect_info_ns_1@10.112.205.101_20220302-065756 %     Steps to reproduce: Create a 2 node cluster with following services kv:n1ql-index. Set index storage mode Enable TLS ./couchbase-cli setting-autofailover -c localhost: 8091 -u Administrator -p password --enable-auto-failover= 0 ./couchbase-cli node-to-node-encryption -c localhost: 8091 -u Administrator -p password --enable ./couchbase-cli setting-security -c localhost: 8091 -u Administrator -p password --set --cluster-encryption-level all ./couchbase-cli setting-autofailover -c localhost: 8091 -u Administrator -p password --enable-auto-failover= 1 --auto-failover-timeout= 120 --max-failovers= 1 ./couchbase-cli setting-security -c https: //localhost:18091 -u Administrator -p password --set --cluster-encryption-level strict --no-ssl-verify Load travel-sample bucket. Run cbcollect_info on both the nodes - /opt/couchbase/bin/cbcollect_info  /tmp/$(hostname).zip I've added the cbcollect zips for reference.
            hemant.rajput Hemant Rajput made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            hemant.rajput Hemant Rajput made changes -
            Attachment node1-cb660-centos7.vagrants.zip [ 178622 ]
            Attachment node2-cb660-centos7.vagrants.zip [ 178623 ]

            Bryan McCoid , From https://review.couchbase.org/c/ns_server/+/171549, I see that we are querying REST using HTTPS protocol but not passing cacert to the REST endpoint. I think this is the reason for the certificate problems. Assigning this ticket to you for further analysis.

            varun.velamuri Varun Velamuri added a comment - Bryan McCoid , From https://review.couchbase.org/c/ns_server/+/171549, I see that we are querying REST using HTTPS protocol but not passing cacert to the REST endpoint. I think this is the reason for the certificate problems. Assigning this ticket to you for further analysis.
            varun.velamuri Varun Velamuri made changes -
            Assignee Hemant Rajput [ hemant.rajput ] Bryan McCoid [ JIRAUSER25453 ]
            bryan.mccoid Bryan McCoid added a comment - - edited

            Just added the -k option to the curl task so we don't need any cert nonsense. I mistakenly thought we had it enabled already (we already had the uppercase -K option, not lowercase k..) but I verified this now with your reproduction steps and I get the dumps for projector as expected in the output. Let me know if it works for you. Thanks

            bryan.mccoid Bryan McCoid added a comment - - edited Just added the -k option to the curl task so we don't need any cert nonsense. I mistakenly thought we had it enabled already (we already had the uppercase -K option, not lowercase k..) but I verified this now with your reproduction steps and I get the dumps for projector as expected in the output. Let me know if it works for you. Thanks
            bryan.mccoid Bryan McCoid made changes -
            Assignee Bryan McCoid [ JIRAUSER25453 ] Varun Velamuri [ varun.velamuri ]
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]

            Build couchbase-server-7.1.0-2421 contains ns_server commit f8f0f11 with commit message:
            MB-51197: add curl '-k' (allow-insecure) option by default to get

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2421 contains ns_server commit f8f0f11 with commit message: MB-51197 : add curl '-k' (allow-insecure) option by default to get
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Assignee Varun Velamuri [ varun.velamuri ] Hemant Rajput [ hemant.rajput ]

            Build couchbase-server-7.2.0-1001 contains ns_server commit f8f0f11 with commit message:
            MB-51197: add curl '-k' (allow-insecure) option by default to get

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-1001 contains ns_server commit f8f0f11 with commit message: MB-51197 : add curl '-k' (allow-insecure) option by default to get

            Build couchbase-server-7.2.0-1001 contains ns_server commit 9d84daa with commit message:
            MB-51197: Add tls status of node to dump-guts/cbcollect

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-1001 contains ns_server commit 9d84daa with commit message: MB-51197 : Add tls status of node to dump-guts/cbcollect

            Validated on build 7.1.0-2445

             

            lfc@LFCs-MacBook-Pro cbcollect_info_ns_1@10.112.205.101_20220308-083859 % cat projector_cprof.log
            ??S?oWfo???%'?x;?z??2$x?̙??? ??UeΜsfw?ݙ?̮??t?n%%?Pe??-R?yi(H,O?x??KEđ?("0:?^;O????~???}??͙????‡?^?]??AkeP??/????F????????uPP??KŦT?6+??P??Ҙ;@V?j4!(?\?TlV@	~?@?*?yu>??ԫ??2???eXSky????6?U???{.?
                                                                                                                                                                                                        y?????N?Ys?b?7?????2,@9d?R(?ʕf̩?v/?Z??
            yG.?ˍS??k?(?ܛU?8??p??@?Y@????<???צ?yxZ=?[ͫ???V???\QWKj??r?NV?k%0_? (@!DQ????ՠ?)???y??uU??I?M%(&????u???Ve(&???R???z=?)?????s~|????Z?A1??7sV?IԟF?VU(?{?M?;?O?g4(?w????7/H?g?:??b??o?xO??Z?P?B??┿??nh
            X?br??9;~?{?}^S?"?7????D??)`	?????n??
                                                    ???W^??;?Oi
            8???sV?+Q??V??rv???E?M?????GS??D]??`
            d??7??A/a???o?s?v?a?g?f;1????(v??v?|3N"?ōv?WӃ??t?n?????D?iB?v??? #??,?1?{??3??l??xy?tk???f????z?KOf*??7I?y???YBF?6????=??	??ߦ3_???YX؞?X?ΈF1?xuV??gC?φ?n?g?d0|??Xn?UN?I???????5#K
                                                                                                                                                                                                  ???e???6??Y.?O??,?m?#??ذ#K??|?Ih?c???-?0?( ??? $???Ɇ??=+S<8??Ѩtx?6?6?	??????e?3?vm?G<X??#??R?N??[^?????e3%???8~??y_?}??%TN?OZd?N??m?E
                                                                                    ??eGw8e??B????N??G??D??
                                                                                                           ?g'>?,??????6???ɀ?Y??C?[??
            SrT??8=MR~T??y??:=o?f?ȋ|??:.?F-????"??aD??⓶9J?M9???G?`z??iV?z?sd{vH[!?-dy                                                ;)',?+<l1???
            Y?9^?{
                  Q??ϙG????-˺m2=$???"?#?=??,h?G?k?a?zx?|?U{h?F?G?sD??0??.?[??Z??,<??6˒Wp????|FI??????\w'?	%                                                                                                                                                    lfc@LFCs-MacBook-Pro cbcollect_info_ns_1@10.112.205.101_20220308-083859 % 

            hemant.rajput Hemant Rajput added a comment - Validated on build 7.1.0-2445   lfc@LFCs-MacBook-Pro cbcollect_info_ns_1@10.112.205.101_20220308-083859 % cat projector_cprof.log ??S?oWfo???%'?x;?z??2$x?̙??? ??UeΜsfw?ݙ?̮??t?n%%?Pe??-R?yi(H,O?x??KEđ?("0:?^;O????~???}??͙????‡?^?]??AkeP??/????F????????uPP??KŦT?6+??P??Ҙ;@V?j4!(?\?TlV@ ~?@?*?yu>??ԫ??2???eXSky????6?U???{.?                                                                                                                                                                                             y?????N?Ys?b?7?????2,@9d?R(?ʕf̩?v/?Z?? yG.?ˍS??k?(?ܛU?8??p??@?Y@????<???צ?yxZ=?[ͫ???V???\QWKj??r?NV?k%0_? (@!DQ????ՠ?)???y??uU??I?M%(&????u???Ve(&???R???z=?)?????s~|????Z?A1??7sV?IԟF?VU(?{?M?;?O?g4(?w????7/H?g?:??b??o?xO??Z?P?B??┿??nh X?br??9;~?{?}^S?"?7????D??)` ?????n??                                         ???W^??;?Oi 8???sV?+Q??V??rv???E?M?????GS??D]??` d??7??A/a???o?s?v?a?g?f;1????(v??v?|3N"?ōv?WӃ??t?n?????D?iB?v??? #??,?1?{??3??l??xy?tk???f????z?KOf*??7I?y???YBF?6????=?? ??ߦ3_???YX؞?X?ΈF1?xuV??gC?φ?n?g?d0|??Xn?UN?I???????5#K                                                                                                                                                                                       ???e???6??Y.?O??,?m?#??ذ#K??|?Ih?c???-?0?( ??? $???Ɇ??=+S<8??Ѩtx?6?6? ??????e?3?vm?G<X??#??R?N??[^?????e3%???8~??y_?}??%TN?OZd?N??m?E                                                                         ??eGw8e??B????N??G??D??                                                                                                ?g'>?,??????6???ɀ?Y??C?[?? SrT??8=MR~T??y??:=o?f?ȋ|??:.?F-????"??aD??⓶9J?M9???G?`z??iV?z?sd{vH[!?-dy                                                ;)',?+<l1??? Y?9^?{       Q??ϙG????-˺m2=$???"?#?=??,h?G?k?a?zx?|?U{h?F?G?sD??0??.?[??Z??,<??6˒Wp????|FI??????\w'? %                                                                                                                                                    lfc@LFCs-MacBook-Pro cbcollect_info_ns_1@10.112.205.101_20220308-083859 % 
            hemant.rajput Hemant Rajput made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            varun.velamuri Varun Velamuri made changes -
            Link This issue is cloned by MB-51752 [ MB-51752 ]
            wayne Wayne Siu made changes -
            Link This issue backports to MB-51752 [ MB-51752 ]
            wayne Wayne Siu made changes -
            Link This issue is cloned by MB-51752 [ MB-51752 ]

            People

              hemant.rajput Hemant Rajput
              varun.velamuri Varun Velamuri
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty