Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25940

Observed DCP backfilling task temporarily suspended because the current memory usage is too high due to which index build hangs

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.0.0
    • 5.0.0
    • couchbase-bucket
    • None
    • Build: 5.0.0-3509

      Cluster Setup:
      Node 1: kv, query (172.23.105.122)
      Node 2: index (172.23.105.163)

    Description

      Steps:

      1. Create and configure the cluster
      2. Create a bucket `travel-sample`. There are 31591 documents and 10 indexes already created on this bucket.
      3. Start creating indexes one by one on this bucket.

      Following are the indexes created and in that order

       

      1. CREATE INDEX def_airportname1 on `travel-sample`(airportname)
      		 
      2. CREATE INDEX def_city1 on `travel-sample`(city)
      		 
      3. CREATE INDEX def_faa1 on `travel-sample`(faa)
      		 
      4. CREATE INDEX def_icao1 on `travel-sample`(icao)
      		 
      5. CREATE INDEX def_name1 on `travel-sample`(name)
      		 
      6. CREATE INDEX def_route_src_dst_day1 on `travel-sample`(sourceairport, destinationairport, (distinct (array (v.day) for v in schedule end)))
      		 
      7. CREATE INDEX def_sourceairport1 on `travel-sample`(sourceairport)
      		 
      8. CREATE INDEX def_type1 on `travel-sample`(type)
      		 
      9. CREATE PRIMARY INDEX p1 on `travel-sample`
      		 
      10. CREATE PRIMARY INDEX p2 on `travel-sample`
      		 
      11. CREATE INDEX def_airportname2 on `travel-sample`(airportname)
      		 
      12. CREATE INDEX def_airportname3 on `travel-sample`(airportname)

       

      def_airportname3 creation hangs at 51%

       

      Around same type memcached logs have following messages:

       

      2017-09-06T01:22:22.161447-07:00 NOTICE (travel-sample) DCP backfilling task temporarily suspended because the current memory usage is too high
      		 
      2017-09-06T01:22:22.162085-07:00 NOTICE 53: (travel-sample) DCP (Producer) eq_dcpq:secidx:proj-travel-sample-INIT_STREAM_TOPIC_73053335a386428452268fd9d3b442c9-8394520178715469292/2 - (vb 530) Creating stream with start seqno 0 and end seqno 18446744073709551615; requested end seqno was 18446744073709551615
      		 
      2017-09-06T01:22:22.162105-07:00 NOTICE 53: (travel-sample) DCP (Producer) eq_dcpq:secidx:proj-travel-sample-INIT_STREAM_TOPIC_73053335a386428452268fd9d3b442c9-8394520178715469292/2 - (vb 530) Scheduling backfill from 1 to 39, reschedule flag : False
      		 
      2017-09-06T01:22:22.162183-07:00 NOTICE 53: (travel-sample) DCP (Producer) eq_dcpq:secidx:proj-travel-sample-INIT_STREAM_TOPIC_73053335a386428452268fd9d3b442c9-8394520178715469292/2 - (vb 530) Sending disk snapshot with start seqno 0 and end seqno 39
      		 
      2017-09-06T01:22:22.162465-07:00 NOTICE (travel-sample) DCP backfilling task temporarily suspended because the current memory usage is too high
      		 
      2017-09-06T01:22:22.163257-07:00 NOTICE 54: (travel-sample) DCP (Producer) eq_dcpq:secidx:proj-travel-sample-INIT_STREAM_TOPIC_73053335a386428452268fd9d3b442c9-8394520178715469292/3 - (vb 531) Creating stream with start seqno 0 and end seqno 18446744073709551615; requested end seqno was 18446744073709551615
      		 
      2017-09-06T01:22:22.163278-07:00 NOTICE 54: (travel-sample) DCP (Producer) eq_dcpq:secidx:proj-travel-sample-INIT_STREAM_TOPIC_73053335a386428452268fd9d3b442c9-8394520178715469292/3 - (vb 531) Scheduling backfill from 1 to 35, reschedule flag : False
      		 
      2017-09-06T01:22:22.164001-07:00 NOTICE (travel-sample) DCP backfilling task temporarily suspended because the current memory usage is too high
      		 
      2017-09-06T01:22:22.164392-07:00 NOTICE 51: (travel-sample) DCP (Producer) eq_dcpq:secidx:proj-travel-sample-INIT_STREAM_TOPIC_73053335a386428452268fd9d3b442c9-8394520178715469292/0 - (vb 532) Creating stream with start seqno 0 and end seqno 18446744073709551615; requested end seqno was 18446744073709551615
      		 
      2017-09-06T01:22:22.164412-07:00 NOTICE 51: (travel-sample) DCP (Producer) eq_dcpq:secidx:proj-travel-sample-INIT_STREAM_TOPIC_73053335a386428452268fd9d3b442c9-8394520178715469292/0 - (vb 532) Scheduling backfill from 1 to 36, reschedule flag : False
      		 
      2017-09-06T01:22:22.164443-07:00 NOTICE (travel-sample) DCP backfilling task temporarily suspended because the current memory usage is too high
      		 
      2017-09-06T01:22:22.165506-07:00 NOTICE 52: (travel-sample) DCP (Producer) eq_dcpq:secidx:proj-travel-sample-INIT_STREAM_TOPIC_73053335a386428452268fd9d3b442c9-8394520178715469292/1 - (vb 533) Creating stream with start seqno 0 and end seqno 18446744073709551615; requested end seqno was 18446744073709551615

       

       

      Similar behaviour is observed on Plasma and MOI as storage mode.

       

      This issue is not observed in build 5.0.0-3497/3506

       

       

       

      Attachments

        For Gerrit Dashboard: MB-25940
        # Subject Branch Project Status CR V
        83140,4 MB-25940: Correct memory accounting for failover callback spock kv_engine Status: MERGED +2 +1
        83146,2 MB-25940: Advance kv_engine in 5.0.0.xml master manifest Status: MERGED +2 +1
        83166,3 Merge remote-tracking branch 'couchbase/spock' master kv_engine Status: MERGED +2 +1

        Activity

          People

            prasanna.gholap Prasanna Gholap [X] (Inactive)
            prasanna.gholap Prasanna Gholap [X] (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty