Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45061

REST api responds slowly in windows boxes

    XMLWordPrintable

Details

    • Untriaged
    • Windows 64-bit
    • 1
    • No

    Description

      Below you can see it responds back in 2.5 mins, I ve seen it go upto 3.5 mins on window boxes.

      Balakumarans-MacBook-Pro-2:cbcollect_info_ns_1@172.23.123.24_20210318-043539 balakumaran.g$ date;curl -u Administrator:password http://172.23.136.108:8091/pools/default;date
      Thu Mar 18 13:23:27 IST 2021
      {"name":"default","nodes":[{"clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@172.23.136.108","thisNode":true,"hostname":"172.23.136.108:8091","nodeUUID":"e2a4c206132916917b5c0abe88379458","clusterCompatibility":458752,"version":"7.0.0-4723-enterprise","os":"win64","cpuCount":6,"ports":{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["kv"],"nodeEncryption":false,"addressFamilyOnly":false,"configuredHostname":"172.23.136.108:8091","addressFamily":"inet","externalListeners":[{"afamily":"inet","nodeEncryption":false}],"couchApiBase":"http://172.23.136.108:8092/","couchApiBaseHTTPS":"https://172.23.136.108:18092/","systemStats":{"cpu_utilization_rate":24.05493699579972,"cpu_stolen_rate":0,"swap_total":7511588864,"swap_used":3617603584,"mem_total":6437847040,"mem_free":3126759424,"mem_limit":6437847040,"cpu_cores_available":6,"allocstall":18446744073709552000},"interestingStats":{"couch_docs_actual_disk_size":149454657,"couch_views_actual_disk_size":0,"curr_items":136192,"curr_items_tot":428400,"ep_bg_fetched":0,"couch_docs_data_size":103947642,"mem_used":218719224,"vb_replica_curr_items":292208,"vb_active_num_non_resident":0,"cmd_get":0,"get_hits":0,"ops":0,"couch_spatial_disk_size":0,"couch_docs_data_size":0,"couch_views_data_size":0,"couch_spatial_data_size":0},"uptime":"629","memoryTotal":6437847040,"memoryFree":3126759424,"mcdMemoryReserved":4911,"mcdMemoryAllocated":4911},{"clusterMembership":"inactiveFailed","recoveryType":"none","status":"healthy","otpNode":"ns_1@172.23.136.114","hostname":"172.23.136.114:8091","nodeUUID":"9f5fdea6d0f7ac4a59c3bd737193b4ad","clusterCompatibility":458752,"version":"7.0.0-4723-enterprise","os":"win64","cpuCount":6,"ports":{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["kv"],"nodeEncryption":false,"addressFamilyOnly":false,"configuredHostname":"172.23.136.114:8091","addressFamily":"inet","externalListeners":[{"afamily":"inet","nodeEncryption":false}],"couchApiBase":"http://172.23.136.114:8092/","couchApiBaseHTTPS":"https://172.23.136.114:18092/","systemStats":{"cpu_utilization_rate":13.17333333333333,"cpu_stolen_rate":0,"swap_total":7511588864,"swap_used":3478118400,"mem_total":6437847040,"mem_free":3372949504,"mem_limit":6437847040,"cpu_cores_available":6,"allocstall":18446744073709552000},"interestingStats":{},"uptime":"621","memoryTotal":6437847040,"memoryFree":3372949504,"mcdMemoryReserved":4911,"mcdMemoryAllocated":4911},{"clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@172.23.136.115","hostname":"172.23.136.115:8091","nodeUUID":"57e94e67161ae81726af21dd12834d4b","clusterCompatibility":458752,"version":"7.0.0-4723-enterprise","os":"win64","cpuCount":6,"ports":{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["kv"],"nodeEncryption":false,"addressFamilyOnly":false,"configuredHostname":"172.23.136.115:8091","addressFamily":"inet","externalListeners":[{"afamily":"inet","nodeEncryption":false}],"couchApiBase":"http://172.23.136.115:8092/","couchApiBaseHTTPS":"https://172.23.136.115:18092/","systemStats":{"cpu_utilization_rate":16.82339259013125,"cpu_stolen_rate":0,"swap_total":7511588864,"swap_used":3722809344,"mem_total":6437847040,"mem_free":3100635136,"mem_limit":6437847040,"cpu_cores_available":6,"allocstall":18446744073709552000},"interestingStats":{"couch_docs_actual_disk_size":143725748,"couch_views_actual_disk_size":0,"curr_items":137312,"curr_items_tot":431648,"ep_bg_fetched":0,"couch_docs_data_size":104265692,"mem_used":216958504,"vb_replica_curr_items":294336,"vb_active_num_non_resident":0,"cmd_get":0,"get_hits":0,"ops":0,"couch_spatial_disk_size":0,"couch_docs_data_size":0,"couch_views_data_size":0,"couch_spatial_data_size":0},"uptime":"571","memoryTotal":6437847040,"memoryFree":3100635136,"mcdMemoryReserved":4911,"mcdMemoryAllocated":4911},{"clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@172.23.136.116","hostname":"172.23.136.116:8091","nodeUUID":"9554998c337f46c19d3cf32d745dca1e","clusterCompatibility":458752,"version":"7.0.0-4723-enterprise","os":"win64","cpuCount":6,"ports":{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["kv"],"nodeEncryption":false,"addressFamilyOnly":false,"configuredHostname":"172.23.136.116:8091","addressFamily":"inet","externalListeners":[{"afamily":"inet","nodeEncryption":false}],"couchApiBase":"http://172.23.136.116:8092/","couchApiBaseHTTPS":"https://172.23.136.116:18092/","systemStats":{"cpu_utilization_rate":7.287152476831789,"cpu_stolen_rate":0,"swap_total":7511588864,"swap_used":3415404544,"mem_total":6437847040,"mem_free":3204485120,"mem_limit":6437847040,"cpu_cores_available":6,"allocstall":18446744073709552000},"interestingStats":{"couch_docs_actual_disk_size":145001757,"couch_views_actual_disk_size":0,"curr_items":135744,"curr_items_tot":429296,"ep_bg_fetched":0,"couch_docs_data_size":103876114,"mem_used":215966576,"vb_replica_curr_items":293552,"vb_active_num_non_resident":0,"cmd_get":0,"get_hits":0,"ops":0,"couch_spatial_disk_size":0,"couch_docs_data_size":0,"couch_views_data_size":0,"couch_spatial_data_size":0},"uptime":"731","memoryTotal":6437847040,"memoryFree":3204485120,"mcdMemoryReserved":4911,"mcdMemoryAllocated":4911},{"clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@172.23.136.117","hostname":"172.23.136.117:8091","nodeUUID":"6d687d1d80ec542a3f1597eb65c66e98","clusterCompatibility":458752,"version":"7.0.0-4723-enterprise","os":"win64","cpuCount":6,"ports":{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["kv"],"nodeEncryption":false,"addressFamilyOnly":false,"configuredHostname":"172.23.136.117:8091","addressFamily":"inet","externalListeners":[{"afamily":"inet","nodeEncryption":false}],"couchApiBase":"http://172.23.136.117:8092/","couchApiBaseHTTPS":"https://172.23.136.117:18092/","systemStats":{"cpu_utilization_rate":19.17063902130071,"cpu_stolen_rate":0,"swap_total":7511588864,"swap_used":3758178304,"mem_total":6437847040,"mem_free":2976632832,"mem_limit":6437847040,"cpu_cores_available":6,"allocstall":18446744073709552000},"interestingStats":{"couch_docs_actual_disk_size":144239066,"couch_views_actual_disk_size":0,"curr_items":128352,"curr_items_tot":432208,"ep_bg_fetched":0,"couch_docs_data_size":104336470,"mem_used":216881352,"vb_replica_curr_items":303856,"vb_active_num_non_resident":0,"cmd_get":0,"get_hits":0,"ops":0,"couch_spatial_disk_size":0,"couch_docs_data_size":0,"couch_views_data_size":0,"couch_spatial_data_size":0},"uptime":"676","memoryTotal":6437847040,"memoryFree":2976632832,"mcdMemoryReserved":4911,"mcdMemoryAllocated":4911}],"buckets":{"uri":"/pools/default/buckets?v=27107170&uuid=13478693bbed0e28da45c902ec10421d","terseBucketsBase":"/pools/default/b/","terseStreamingBucketsBase":"/pools/default/bs/"},"remoteClusters":{"uri":"/pools/default/remoteClusters?uuid=13478693bbed0e28da45c902ec10421d","validateURI":"/pools/default/remoteClusters?just_validate=1"},"alerts":[],"alertsSilenceURL":"/controller/resetAlerts?uuid=13478693bbed0e28da45c902ec10421d&token=0","controllers":{"addNode":{"uri":"/controller/addNodeV2?uuid=13478693bbed0e28da45c902ec10421d"},"rebalance":{"uri":"/controller/rebalance?uuid=13478693bbed0e28da45c902ec10421d"},"failOver":{"uri":"/controller/failOver?uuid=13478693bbed0e28da45c902ec10421d"},"startGracefulFailover":{"uri":"/controller/startGracefulFailover?uuid=13478693bbed0e28da45c902ec10421d"},"reAddNode":{"uri":"/controller/reAddNode?uuid=13478693bbed0e28da45c902ec10421d"},"reFailOver":{"uri":"/controller/reFailOver?uuid=13478693bbed0e28da45c902ec10421d"},"ejectNode":{"uri":"/controller/ejectNode?uuid=13478693bbed0e28da45c902ec10421d"},"setRecoveryType":{"uri":"/controller/setRecoveryType?uuid=13478693bbed0e28da45c902ec10421d"},"setAutoCompaction":{"uri":"/controller/setAutoCompaction?uuid=13478693bbed0e28da45c902ec10421d","validateURI":"/controller/setAutoCompaction?just_validate=1"},"clusterLogsCollection":{"startURI":"/controller/startLogsCollection?uuid=13478693bbed0e28da45c902ec10421d","cancelURI":"/controller/cancelLogsCollection?uuid=13478693bbed0e28da45c902ec10421d"},"replication":{"createURI":"/controller/createReplication?uuid=13478693bbed0e28da45c902ec10421d","validateURI":"/controller/createReplication?just_validate=1"}},"rebalanceStatus":"none","rebalanceProgressUri":"/pools/default/rebalanceProgress","stopRebalanceUri":"/controller/stopRebalance?uuid=13478693bbed0e28da45c902ec10421d","nodeStatusesUri":"/nodeStatuses","maxBucketCount":30,"maxCollectionCount":1200,"maxScopeCount":1200,"autoCompactionSettings":{"parallelDBAndViewCompaction":false,"databaseFragmentationThreshold":{"percentage":30,"size":"undefined"},"viewFragmentationThreshold":{"percentage":30,"size":"undefined"},"indexCompactionMode":"circular","indexCircularCompaction":{"daysOfWeek":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","interval":{"fromHour":0,"toHour":0,"fromMinute":0,"toMinute":0,"abortOutside":false}},"indexFragmentationThreshold":{"percentage":30}},"tasks":{"uri":"/pools/default/tasks?v=108545295"},"counters":{"graceful_failover_success":1,"failover":1,"failover_complete":1,"graceful_failover_start":1,"rebalance_success":1,"rebalance_start":1},"indexStatusURI":"/indexStatus?v=21137658","serverGroupsUri":"/pools/default/serverGroups?v=120999064","clusterName":"","balanced":false,"checkPermissionsURI":"/pools/default/checkPermissions","memoryQuota":4329,"indexMemoryQuota":512,"ftsMemoryQuota":512,"cbasMemoryQuota":1392,"eventingMemoryQuota":256,"storageTotals":{"ram":{"total":25751388160,"quotaTotal":18157142016,"quotaUsed":7130316800,"used":13227814912,"usedByData":868525656,"quotaUsedPerNode":1782579200,"quotaTotalPerNode":4539285504},"hdd":{"total":298542153728,"quotaTotal":298542153728,"used":114192373800,"usedByData":582421228,"free":182110713776}}}Thu Mar 18 13:25:49 IST 2021
      

      cbcollect_info attached.

      Attachments

        1. CLOSE_WAIT_with_owner_node_237.txt
          4 kB
        2. collectinfo-2021-03-18T080926-ns_1@172.23.136.108.zip
          38.94 MB
        3. consoleText_12GB_12_Core.txt
          41 kB
        4. erl_crash.dump
          22.02 MB
        5. image-2021-04-19-11-07-42-387.png
          image-2021-04-19-11-07-42-387.png
          178 kB
        6. image-2021-04-19-11-09-35-035.png
          image-2021-04-19-11-09-35-035.png
          297 kB
        7. image-2021-04-26-11-52-26-733.png
          image-2021-04-26-11-52-26-733.png
          276 kB
        8. MB-45061_auto.txt
          24 kB
        9. most-recent.tgz
          24.03 MB
        10. ns_server_debug_log_fragment.txt
          1 kB
        11. ns_server_error_log_fragment.txt
          203 kB
        12. ns_server_http_access_log_fragment.txt
          1 kB
        13. ns_server_info_log_fragment.txt
          835 kB
        14. processes.txt
          53 kB
        15. Promtimer_node_237_no_cpu_usage_no_free_mem.png
          Promtimer_node_237_no_cpu_usage_no_free_mem.png
          105 kB
        16. screenshot-1.png
          screenshot-1.png
          42 kB
        17. screenshot-10.png
          screenshot-10.png
          340 kB
        18. screenshot-11.png
          screenshot-11.png
          529 kB
        19. screenshot-12.png
          screenshot-12.png
          107 kB
        20. screenshot-13.png
          screenshot-13.png
          276 kB
        21. screenshot-14.png
          screenshot-14.png
          753 kB
        22. screenshot-15.png
          screenshot-15.png
          554 kB
        23. screenshot-16.png
          screenshot-16.png
          100 kB
        24. screenshot-17.png
          screenshot-17.png
          665 kB
        25. screenshot-18.png
          screenshot-18.png
          110 kB
        26. screenshot-19.png
          screenshot-19.png
          111 kB
        27. screenshot-2.png
          screenshot-2.png
          78 kB
        28. screenshot-20.png
          screenshot-20.png
          97 kB
        29. Screenshot 2021-04-20 at 8.36.48 AM.png
          Screenshot 2021-04-20 at 8.36.48 AM.png
          1.02 MB
        30. Screenshot 2021-04-20 at 8.36.48 AM-1.png
          Screenshot 2021-04-20 at 8.36.48 AM-1.png
          1.02 MB
        31. screenshot-21.png
          screenshot-21.png
          602 kB
        32. screenshot-22.png
          screenshot-22.png
          111 kB
        33. screenshot-23.png
          screenshot-23.png
          102 kB
        34. screenshot-24.png
          screenshot-24.png
          244 kB
        35. screenshot-25.png
          screenshot-25.png
          80 kB
        36. screenshot-26.png
          screenshot-26.png
          2.15 MB
        37. screenshot-27.png
          screenshot-27.png
          282 kB
        38. screenshot-28.png
          screenshot-28.png
          84 kB
        39. screenshot-29.png
          screenshot-29.png
          487 kB
        40. screenshot-3.png
          screenshot-3.png
          81 kB
        41. screenshot-4.png
          screenshot-4.png
          76 kB
        42. screenshot-5.png
          screenshot-5.png
          89 kB
        43. screenshot-6.png
          screenshot-6.png
          81 kB
        44. screenshot-7.png
          screenshot-7.png
          79 kB
        45. screenshot-8.png
          screenshot-8.png
          102 kB
        46. screenshot-9.png
          screenshot-9.png
          108 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            I manage to get a crash dump during the time memory ramped up erl_crash.dump

            And again, logger_simple_h process shows up at the top with a very large queue and memory:

            meni.hillel Meni Hillel (Inactive) added a comment - I manage to get a crash dump during the time memory ramped up erl_crash.dump And again, logger_simple_h process shows up at the top with a very large queue and memory:

            Meni Hillel - Since the bug is resolved, can we reclaim those boxes?

            cc : Ashwin Govindarajulu

            Balakumaran.Gopal Balakumaran Gopal added a comment - Meni Hillel - Since the bug is resolved, can we reclaim those boxes? cc : Ashwin Govindarajulu

            I think we pretty much zoomed in on "excessive logging" as root cause for this issue and are close to conclusion of the investigation.

            Yet we still don't have a compelling explanation for the differences between 6.6 and 7.0.

            Aliaksey Artamonau Aliaksey Artamonau (Inactive) added a comment - I think we pretty much zoomed in on "excessive logging" as root cause for this issue and are close to conclusion of the investigation. Yet we still don't have a compelling explanation for the differences between 6.6 and 7.0.

            Balakumaran Gopal We would like to hold on to the system as part of the work done for the related ticket MB-45793.

            meni.hillel Meni Hillel (Inactive) added a comment - Balakumaran Gopal We would like to hold on to the system as part of the work done for the related ticket MB-45793 .

            Failover + rebalance runs looks good on 7.0.0-5017.
            Logs : http://greenboard.sc.couchbase.com/#!/server/7.0.0/5017?platforms=WIN&features=COLLECTIONS

            Marking this closed.

            Balakumaran.Gopal Balakumaran Gopal added a comment - Failover + rebalance runs looks good on 7.0.0-5017. Logs : http://greenboard.sc.couchbase.com/#!/server/7.0.0/5017?platforms=WIN&features=COLLECTIONS Marking this closed.

            People

              dfinlay Dave Finlay
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty