Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60234

Memory violations seen during system test run

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown

    Description

      I'm still seeing replica violation messages with system test run on 7.6.0-1969 -

      Seen on 106.171 -

       --- Violations for index <idx4_I8HTINCX6P_idxprefix 1, bucket4, _default, Zlkooz3nbhr0d2w5anTGOTTPOS0TTh> (mem 0, cpu 0) at node 172.23.97.109:8091 
      	Cannot move to 172.23.106.30:8091: MemoryViolation (free mem 61.1761M, free cpu -3.2854434179315)
      		 
      	Cannot move to 172.23.105.122:8091: MemoryViolation (free mem 453.08M, free cpu -3.3821407839108915)
      		 
      	Cannot move to 172.23.106.171:8091: ReplicaViolation (free mem 1.67772e+07T, free cpu -3.2465831893210915)
      		 
      	Cannot move to 172.23.97.108:8091: MemoryViolation (free mem 109.256M, free cpu 0.9526508836278698)
      		 
      --- Violations for index <idx4_I8HTINCX6P_idxprefix 3, bucket4, _default, Zlkooz3nbhr0d2w5anTGOTTPOS0TTh> (mem 0, cpu 0) at node 172.23.97.109:8091 
      	Cannot move to 172.23.106.30:8091: MemoryViolation (free mem 61.1761M, free cpu -3.2854434179315)
      		 
      	Cannot move to 172.23.105.122:8091: MemoryViolation (free mem 453.08M, free cpu -3.3821407839108915)
      		 
      	Cannot move to 172.23.106.171:8091: MemoryViolation (free mem 1.67772e+07T, free cpu -3.2465831893210915)
      		 
      	Cannot move to 172.23.97.108:8091: MemoryViolation (free mem 109.256M, free cpu 0.9526508836278698)
      		 
      --- Violations for index <idx4_I8HTINCX6P_idxprefix 4, bucket4, _default, Zlkooz3nbhr0d2w5anTGOTTPOS0TTh> (mem 0, cpu 0) at node 172.23.97.109:8091 
      	Cannot move to 172.23.106.30:8091: MemoryViolation (free mem 61.1761M, free cpu -3.2854434179315)
      		 
      	Cannot move to 172.23.105.122:8091: MemoryViolation (free mem 453.08M, free cpu -3.3821407839108915)
      		 
      	Cannot move to 172.23.106.171:8091: ReplicaViolation (free mem 1.67772e+07T, free cpu -3.2465831893210915)
      		 
      	Cannot move to 172.23.97.108:8091: MemoryViolation (free mem 109.256M, free cpu 0.9526508836278698)
      		 
      --- Violations for index <idx4_I8HTINCX6P_idxprefix 3 (replica 1), bucket4, _default, Zlkooz3nbhr0d2w5anTGOTTPOS0TTh> (mem 0, cpu 0) at node 172.23.97.109:8091 
      	Cannot move to 172.23.106.30:8091: MemoryViolation (free mem 61.1761M, free cpu -3.2854434179315)
      		 
      	Cannot move to 172.23.105.122:8091: MemoryViolation (free mem 453.08M, free cpu -3.3821407839108915)
      		 
      	Cannot move to 172.23.106.171:8091: MemoryViolation (free mem 1.67772e+07T, free cpu -3.2465831893210915)
      		 
      	Cannot move to 172.23.97.108:8091: MemoryViolation (free mem 109.256M, free cpu 0.9526508836278698)
      		 
      --- Violations for index <idx4_I8HTINCX6P_idxprefix 2, bucket4, _default, Zlkooz3nbhr0d2w5anTGOTTPOS0TTh> (mem 0, cpu 0) at node 172.23.105.122:8091 
      	Cannot move to 172.23.106.30:8091: MemoryViolation (free mem 61.1761M, free cpu -3.2854434179315)
      		 
      	Cannot move to 172.23.97.109:8091: MemoryViolation (free mem 579.205M, free cpu -3.3443705529051986)
      		 
      	Cannot move to 172.23.106.171:8091: ReplicaViolation (free mem 1.67772e+07T, free cpu -3.2465831893210915)
      		 
      	Cannot move to 172.23.97.108:8091: MemoryViolation (free mem 109.256M, free cpu 0.9526508836278698)
      		 
      --- Violations for index <idx4_I8HTINCX6P_idxprefix 5, bucket4, _default, Zlkooz3nbhr0d2w5anTGOTTPOS0TTh> (mem 0, cpu 0) at node 172.23.105.122:8091 
      	Cannot move to 172.23.106.30:8091: MemoryViolation (free mem 61.1761M, free cpu -3.2854434179315)
      		 
      	Cannot move to 172.23.97.109:8091: MemoryViolation (free mem 579.205M, free cpu -3.3443705529051986)
      		 
      	Cannot move to 172.23.106.171:8091: ReplicaViolation (free mem 1.67772e+07T, free cpu -3.2465831893210915)
      		 
      	Cannot move to 172.23.97.108:8091: MemoryViolation (free mem 109.256M, free cpu 0.9526508836278698)
      		 
      --- Violations for index <idx4_I8HTINCX6P_idxprefix 1 (replica 1), bucket4, _default, Zlkooz3nbhr0d2w5anTGOTTPOS0TTh> (mem 0, cpu 0) at node 172.23.106.171:8091 
      	Cannot move to 172.23.106.30:8091: MemoryViolation (free mem 61.1761M, free cpu -3.2854434179315)
      		 
      	Cannot move to 172.23.97.109:8091: ReplicaViolation (free mem 579.205M, free cpu -3.3443705529051986)
      		 
      	Cannot move to 172.23.105.122:8091: MemoryViolation (free mem 453.08M, free cpu -3.3821407839108915)
      		 
      	Cannot move to 172.23.97.108:8091: MemoryViolation (free mem 109.256M, free cpu 0.9526508836278698)
      		 
      --- Violations for index <idx4_I8HTINCX6P_idxprefix 2 (replica 1), bucket4, _default, Zlkooz3nbhr0d2w5anTGOTTPOS0TTh> (mem 0, cpu 0) at node 172.23.106.171:8091 
      	Cannot move to 172.23.106.30:8091: MemoryViolation (free mem 61.1761M, free cpu -3.2854434179315)
      		 
      	Cannot move to 172.23.97.109:8091: MemoryViolation (free mem 579.205M, free cpu -3.3443705529051986)
      		 
      	Cannot move to 172.23.105.122:8091: ReplicaViolation (free mem 453.08M, free cpu -3.3821407839108915)
      		 
      	Cannot move to 172.23.97.108:8091: MemoryViolation (free mem 109.256M, free cpu 0.9526508836278698)
      		 
      --- Violations for index <idx4_I8HTINCX6P_idxprefix 4 (replica 1), bucket4, _default, Zlkooz3nbhr0d2w5anTGOTTPOS0TTh> (mem 0, cpu 0) at node 172.23.106.171:8091 
      	Cannot move to 172.23.106.30:8091: MemoryViolation (free mem 61.1761M, free cpu -3.2854434179315)
      		 
      	Cannot move to 172.23.97.109:8091: ReplicaViolation (free mem 579.205M, free cpu -3.3443705529051986)
      		 
      	Cannot move to 172.23.105.122:8091: MemoryViolation (free mem 453.08M, free cpu -3.3821407839108915)
      		 
      	Cannot move to 172.23.97.108:8091: MemoryViolation (free mem 109.256M, free cpu 0.9526508836278698)
      		 
      --- Violations for index <idx4_I8HTINCX6P_idxprefix 5 (replica 1), bucket4, _default, Zlkooz3nbhr0d2w5anTGOTTPOS0TTh> (mem 0, cpu 0) at node 172.23.106.171:8091 
      	Cannot move to 172.23.106.30:8091: MemoryViolation (free mem 61.1761M, free cpu -3.2854434179315)
      		 
      	Cannot move to 172.23.97.109:8091: MemoryViolation (free mem 579.205M, free cpu -3.3443705529051986)
      		 
      	Cannot move to 172.23.105.122:8091: ReplicaViolation (free mem 453.08M, free cpu -3.3821407839108915)
      		 
      	Cannot move to 172.23.97.108:8091: MemoryViolation (free mem 109.256M, free cpu 0.9526508836278698)
      		 
      2023-12-26T04:10:18.756-08:00 [Error] SAPlanner::Validated Error observed when constraint sizing, err: Total memory

      Cbcollect logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1703593154/collectinfo-2023-12-26T122936-ns_1%40172.23.105.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1703593154/collectinfo-2023-12-26T122936-ns_1%40172.23.106.171.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1703593154/collectinfo-2023-12-26T122936-ns_1%40172.23.106.30.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1703593154/collectinfo-2023-12-26T122936-ns_1%40172.23.96.198.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1703593154/collectinfo-2023-12-26T122936-ns_1%40172.23.96.230.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1703593154/collectinfo-2023-12-26T122936-ns_1%40172.23.96.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1703593154/collectinfo-2023-12-26T122936-ns_1%40172.23.97.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1703593154/collectinfo-2023-12-26T122936-ns_1%40172.23.97.108.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1703593154/collectinfo-2023-12-26T122936-ns_1%40172.23.97.109.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1703593154/collectinfo-2023-12-26T122936-ns_1%40172.23.97.66.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1703593154/collectinfo-2023-12-26T122936-ns_1%40172.23.97.67.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            varun.velamuri Varun Velamuri
            varun.velamuri Varun Velamuri
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty