Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38007

60% rebalance time increase in ephemeral bucket in 7.0.0-1272 or earlier version

    XMLWordPrintable

Details

    • Triaged
    • Yes
    • KV-Engine 2021-March

    Description

      Observing ~30% increase in rebalance time for ephemeral bucket in 7.0.0-1272 build .

       

       

       

      job with degradation : 

      http://perf.jenkins.couchbase.com/job/titan/7016/

      Note : First observed this in 7.0.0-1272 build . Need to triage it further to identify the exact build which introduced possible regression .

      i don't see anything obvious from stats other than 6.5.0-4960 build had higher cpu utilization compared to 7.0.0-1272 

      Attachments

        1. dcp.png
          dcp.png
          31 kB
        2. ephmemeral-rebalance-jan21.png
          ephmemeral-rebalance-jan21.png
          41 kB
        3. image.png
          image.png
          86 kB
        4. image-2020-02-20-13-08-51-643.png
          image-2020-02-20-13-08-51-643.png
          88 kB
        5. image-2020-02-21-12-18-08-330.png
          image-2020-02-21-12-18-08-330.png
          88 kB
        6. image-2021-03-22-10-18-46-159.png
          image-2021-03-22-10-18-46-159.png
          100 kB
        7. image-2021-03-23-21-14-01-015.png
          image-2021-03-23-21-14-01-015.png
          96 kB
        8. Rebalnce-in-ephemeral.png
          Rebalnce-in-ephemeral.png
          35 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            So in the bufferevent world we work very simplified like (lets assume only one socket)

            1. evpoll returns the socket
            2. bufferevent determines if it can read and drain the socket completely and put it all in the bufferevent input queue
            3. bufferevent determines if it can write and then tries to write as much as possible from what we've queued for sending to the socket.
            4. if we have something in the read buffer (or we completed a write) we call the rw_callback (we could have had two independed)
            4.1 Look at all "currently running commands" and see if we can continue on any of them
            4.2 Process up to X commands pending in the input queue (for DCP this would be all of the ack's)
            4.3 For DCP connections try to run step() until the engine blocks or we've queued X MB of data to send
            5. Disable read events if we've got too much data queued for the other end (in the socket buffer) or if we have too many commands currently running.

            Now that I look at it it could be that we would should ignore the queue size for DCP connections as this would cause us to wait until the we've drained the userspace queue before getting a callback for new messages. I'll build up a toy build with that

            trond Trond Norbye added a comment - So in the bufferevent world we work very simplified like (lets assume only one socket) 1. evpoll returns the socket 2. bufferevent determines if it can read and drain the socket completely and put it all in the bufferevent input queue 3. bufferevent determines if it can write and then tries to write as much as possible from what we've queued for sending to the socket. 4. if we have something in the read buffer (or we completed a write) we call the rw_callback (we could have had two independed) 4.1 Look at all "currently running commands" and see if we can continue on any of them 4.2 Process up to X commands pending in the input queue (for DCP this would be all of the ack's) 4.3 For DCP connections try to run step() until the engine blocks or we've queued X MB of data to send 5. Disable read events if we've got too much data queued for the other end (in the socket buffer) or if we have too many commands currently running. Now that I look at it it could be that we would should ignore the queue size for DCP connections as this would cause us to wait until the we've drained the userspace queue before getting a callback for new messages. I'll build up a toy build with that
            drigby Dave Rigby added a comment - - edited

            Tested patch http://review.couchbase.org/c/kv_engine/+/148145 which modifies Connection::executeCommandPipeline to check more frequently for new messages (i.e. DCP buffer ack messages) using dcpdrain command with varying buffer sizes on macOS:

            1. Start single-node with 256 vBuckets (roughly model 4 node cluster), 4GB quota

              COUCHBASE_NUM_VBUCKETS=256 ./cluster_run --nodes=1 
              ./cluster_connect -n1 -s 4096
              

            2. Populate bucket with 10M 64Byte documents:

              cbc-pillowfight -U 127.0.0.1:9000/default -u Administrator -P asdasd --num-items=10000000 -m 64 -M 64 --num-threads 8 --populate-only
              

            3. Compact bucket.
            4. Run dcpdrain and measure throughput / pause times:

              dcpdrain --host localhost:12000 -b default -u Administrator -P asdasd -B <BUFFER SIZE>
              

            Results (median of 3 runs of dcpdrain command for each config):

            Configuration Buffer Size (B) Throughput (MB/s) Total pause time (s) BufferLogFull pause time (s) BufferLogFull count Notes
            Baseline 1024 9.2 67 67 2864080  
            Baseline 10240 34 8.1 8.1 317427  
            Baseline 102400 46 2.2 2.2 31981  
            Baseline 1024000 27 3.3 0 0 [1]
            patch 148145 1024 9.3 69 68 2867480  
            patch 148145 10240 35 8.1 8.1 317434  
            patch 148145 102400 43 2.2 1.8 27656  
            patch 148145 1024000 30 2.6 0 0 [1]

            Of note is at buffer size 1024000 (1) there were zero pauses due to BufferLogFull; the pause time was due to "ReadyListEmpty":

            {"count":246626, "duration":"2649 ms"}

            . There is also a lower throughput at this buffer size.

            Conclusion: Not much change seen with patch; but need to investigate why throughput drops off at larger buffer sizes.

            drigby Dave Rigby added a comment - - edited Tested patch http://review.couchbase.org/c/kv_engine/+/148145 which modifies Connection::executeCommandPipeline to check more frequently for new messages (i.e. DCP buffer ack messages) using dcpdrain command with varying buffer sizes on macOS: Start single-node with 256 vBuckets (roughly model 4 node cluster), 4GB quota COUCHBASE_NUM_VBUCKETS=256 ./cluster_run --nodes=1 ./cluster_connect -n1 -s 4096 Populate bucket with 10M 64Byte documents: cbc-pillowfight -U 127.0.0.1:9000/default -u Administrator -P asdasd --num-items=10000000 -m 64 -M 64 --num-threads 8 --populate-only Compact bucket. Run dcpdrain and measure throughput / pause times: dcpdrain --host localhost:12000 -b default -u Administrator -P asdasd -B <BUFFER SIZE> Results (median of 3 runs of dcpdrain command for each config): Configuration Buffer Size (B) Throughput (MB/s) Total pause time (s) BufferLogFull pause time (s) BufferLogFull count Notes Baseline 1024 9.2 67 67 2864080   Baseline 10240 34 8.1 8.1 317427   Baseline 102400 46 2.2 2.2 31981   Baseline 1024000 27 3.3 0 0 [1] patch 148145 1024 9.3 69 68 2867480   patch 148145 10240 35 8.1 8.1 317434   patch 148145 102400 43 2.2 1.8 27656   patch 148145 1024000 30 2.6 0 0 [1] Of note is at buffer size 1024000 (1) there were zero pauses due to BufferLogFull; the pause time was due to "ReadyListEmpty": {"count":246626, "duration":"2649 ms"} . There is also a lower throughput at this buffer size. Conclusion: Not much change seen with patch; but need to investigate why throughput drops off at larger buffer sizes.
            drigby Dave Rigby added a comment -

            Ran same test on mancouch (Ubuntu 18.04, Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz, 12 cores, 24 hyperthreads)

            Configuration Buffer Size (B) Throughput (MB/s) Total pause time (s) BufferLogFull pause time (s) BufferLogFull count Notes
            Baseline 1024 6.8 151 150 2503424  
            Baseline 10240 23 16 16 540471  
            Baseline 102400 30 5.1 4.7 39352  
            Baseline 1024000 29 4.3 1.6 2631  
            patch 148145 1024 7.6 129 129 2504370  
            patch 148145 10240 23 151 15 540527  
            patch 148145 102400 31 5.1 5.0 41002  
            patch 148145 1024000 32 2.8 2.0 3525  

            There is a small improvement with patch 148145 here, particulary on the smaller buffer sizes - both in terms of Throughput and the amount of time paused.

            drigby Dave Rigby added a comment - Ran same test on mancouch (Ubuntu 18.04, Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz, 12 cores, 24 hyperthreads) Configuration Buffer Size (B) Throughput (MB/s) Total pause time (s) BufferLogFull pause time (s) BufferLogFull count Notes Baseline 1024 6.8 151 150 2503424   Baseline 10240 23 16 16 540471   Baseline 102400 30 5.1 4.7 39352   Baseline 1024000 29 4.3 1.6 2631   patch 148145 1024 7.6 129 129 2504370   patch 148145 10240 23 151 15 540527   patch 148145 102400 31 5.1 5.0 41002   patch 148145 1024000 32 2.8 2.0 3525   There is a small improvement with patch 148145 here, particulary on the smaller buffer sizes - both in terms of Throughput and the amount of time paused.

            Build couchbase-server-7.0.0-4690 contains kv_engine commit dadfa38 with commit message:
            MB-38007: Limit the DCP send queue from step() to 1MB

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4690 contains kv_engine commit dadfa38 with commit message: MB-38007 : Limit the DCP send queue from step() to 1MB

            closing this ticket based on recent performance improvement in 7.0.0-4690 build .

            sharath.sulochana Sharath Sulochana (Inactive) added a comment - closing this ticket based on recent performance improvement in 7.0.0-4690 build .

            People

              sharath.sulochana Sharath Sulochana (Inactive)
              sharath.sulochana Sharath Sulochana (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty