Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29009

Idle indexer makes more than 30 CPU cores busy when there are multiple partitioned indexes

    XMLWordPrintable

Details

    Description

      Build 5.5.0-2326.

      The indexer process uses more than 3,000% CPU on one node with 10 partitioned indexes.

      Setup:

      • 4 KV nodes, 1 index node
      • 1 bucket
      • 100M items
      • Plasma

      Sample document:

      {
        "sold_date": "2000-09-25",
        "item": {
          "id": "AAAAAAAAKGJJBAAA",
          "category": "Men",
          "brand": "importoimporto #1",
          "manufacturer_id": 15
        },
        "wholesale_cost": 63.63,
        "list_price": 101.17,
        "quantity": 37,
        "customer": {
          "zip": "71083",
          "purchase_estimate": 6000,
          "name": " ",
          "credit_rating": "High Risk",
          "gender": "M",
          "birth_country": "",
          "dep_count": 0,
          "education_status": "College",
          "dep_college_count": 0,
          "id": "AAAAAAAAOJOMBBAA",
          "dep_employed_count": 0,
          "preferred_flag": false,
          "state": "TX",
          "marital_status": "S",
          "county": "Armstrong County"
        },
        "coupon_amt": 46.03,
        "store": "able",
        "sales_price": 41.47
      }
      

      Indexes:

          CREATE INDEX ag1 ON `bucket-1`(DATE_PART_STR(sold_date, 'year'), customer.state, store, quantity) PARTITION BY HASH(store) WITH {"defer_build": true};
          CREATE INDEX ag2 ON `bucket-1`(DATE_PART_STR(sold_date, 'year'), DATE_PART_STR(sold_date, 'week'), WEEKDAY_STR(sold_date), sales_price) PARTITION BY HASH(sold_date) WITH {"defer_build": true};
          CREATE INDEX ag3 ON `bucket-1`(item.manufacturer_id, DATE_PART_STR(sold_date, 'month'), DATE_PART_STR(sold_date, 'year'), item.brand, sales_price) PARTITION BY HASH(sold_date) WITH {"defer_build": true};
          CREATE INDEX ag4 ON `bucket-1`(DATE_PART_STR(sold_date, 'year'), customer.preferred_flag, customer.birth_country, wholesale_cost, sales_price) PARTITION BY HASH(customer.birth_country) WITH {"defer_build": true};
          CREATE INDEX ag5 ON `bucket-1`(DATE_PART_STR(sold_date, 'year'), DATE_PART_STR(sold_date, 'month'), sales_price, customer.state) PARTITION BY HASH(customer.state) WITH {"defer_build": true};
          CREATE INDEX ag6 ON `bucket-1`(customer.gender, customer.marital_status, customer.education_status, item.id, quantity, list_price, coupon_amt, sales_price) PARTITION BY HASH(item.id) WITH {"defer_build": true};
          CREATE INDEX ag7 ON `bucket-1`(customer.zip, customer.preferred_flag) PARTITION BY HASH(customer.zip) WITH {"defer_build": true};
          CREATE INDEX ag8 ON `bucket-1`(customer.county, DATE_PART_STR(sold_date, 'year'), DATE_PART_STR(sold_date, 'month'), customer.gender, customer.marital_status, customer.education_status, customer.purchase_estimate, customer.credit_rating, customer.dep_count, customer.dep_employed_count, customer.dep_college_count) PARTITION BY HASH(customer.county) WITH {"defer_build": true};
          CREATE INDEX ag9 ON `bucket-1`(item.category, DATE_PART_STR(sold_date, 'day_of_year'), sales_price) PARTITION BY HASH(item.category) WITH {"defer_build": true};
          CREATE INDEX ag10 ON `bucket-1`(customer.zip, DATE_PART_STR(sold_date, 'year'), DATE_PART_STR(sold_date, 'quarter'), sales_price) PARTITION BY HASH(customer.zip) WITH {"defer_build": true};
          BUILD  INDEX ON `bucket-1`(ag1, ag2, ag3, ag4, ag5, ag6, ag7, ag8, ag9, ag10);
      

      Indexer seems very busy while trying to estimate the amount of memory in use:

      ROUTINE ======================== github.com/couchbase/plasma.(*Plasma).MemoryInUse in /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go
        4.30mins   4.36mins (flat, cum) 93.02% of Total
               .          .   1132:	r.iter.snap = snap
               .          .   1133:	return r.iter, nil
               .          .   1134:}
               .          .   1135:
               .          .   1136:// MemoryInUse returns the memory used by the plasma instance
           140ms      140ms   1137:func (s *Plasma) MemoryInUse() int64 {
               .          .   1138:	var memSz int64
          29.58s     33.13s   1139:	for w := s.getWCtxList(); w != nil; w = w.next {
        2.68mins   2.68mins   1140:		memSz += w.sts.AllocSz - w.sts.FreeSz
        1.13mins   1.13mins   1141:		memSz += w.sts.AllocSzIndex - w.sts.FreeSzIndex
               .          .   1142:	}
               .          .   1143:
            40ms       40ms   1144:	return memSz
               .          .   1145:}
               .          .   1146:
               .          .   1147:// GetStats() returns a stats object
               .          .   1148:func (s *Plasma) GetStats() Stats {
               .          .   1149:	var rdrSts Stats
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pavelpaulau Pavel Paulau (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty