Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37144

KV-Engine NonIO and AuxIO threads incorrectly set to lowest priority

    XMLWordPrintable

Details

    • Triaged
    • Yes
    • KV-Engine Mad-Hatter GA

    Description

      While investigating MB-36765, I observed the NonIO and AuxIO threads are incorrect set to the lowest priority - this should only be the case for Writer threads:

      $ ps -o pid,tid,ni,pri,comm -p $(pgrep memcached|head -n1) -T|grep "reader\|writer\|aux\|non"
      14714 18837   0  19 mc:reader_0
      ...
      14714 18854   0  19 mc:reader_15
      14714 18855   0  19 mc:writer_0
      14714 18856  19   0 mc:writer_1
      14714 18857  19   0 mc:writer_2
      14714 18858  19   0 mc:writer_3
      14714 18859  19   0 mc:auxIO_0
      14714 18860  19   0 mc:auxIO_1
      14714 18861  19   0 mc:auxIO_2
      14714 18862  19   0 mc:nonIO_0
      14714 18863  19   0 mc:nonIO_1
      14714 18864  19   0 mc:nonIO_2
      14714 18865  19   0 mc:nonIO_3
      14714 18866  19   0 mc:nonIO_4
      14714 18867  19   0 mc:nonIO_5
      14714 18868  19   0 mc:nonIO_6
      

      Note priority (3rd column) of the auxIO and nonIO is the same as the writers (19) when it should be the same as the readers (0).

      See http://review.couchbase.org/#/c/117651/ for the patch which introduced the priority change.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            drigby Dave Rigby created issue -
            drigby Dave Rigby made changes -
            Field Original Value New Value
            Priority Major [ 3 ] Critical [ 2 ]
            drigby Dave Rigby made changes -
            Is this a Regression? Unknown [ 10452 ] Yes [ 10450 ]
            drigby Dave Rigby added a comment -

            Note further that the priority of the writer thread themselves is incorrect the first writer thread isn't getting it's priority set:

            14714 18855   0  19 mc:writer_0
            14714 18856  19   0 mc:writer_1
            14714 18857  19   0 mc:writer_2
            14714 18858  19   0 mc:writer_3
            

            drigby Dave Rigby added a comment - Note further that the priority of the writer thread themselves is incorrect the first writer thread isn't getting it's priority set: 14714 18855 0 19 mc:writer_0 14714 18856 19 0 mc:writer_1 14714 18857 19 0 mc:writer_2 14714 18858 19 0 mc:writer_3
            drigby Dave Rigby made changes -
            Description While investigating MB-36785, I observed the NonIO and AuxIO threads are incorrect set to the lowest priority - this should only be the case for Writer threads:

            {code}
            $ ps -o pid,tid,ni,pri,comm -p $(pgrep memcached|head -n1) -T|grep "reader\|writer\|aux\|non"
            14714 18837 0 19 mc:reader_0
            14714 18838 0 19 mc:reader_1
            14714 18839 0 19 mc:reader_2
            14714 18840 0 19 mc:reader_3
            14714 18841 0 19 mc:reader_4
            14714 18842 0 19 mc:reader_5
            14714 18843 0 19 mc:reader_6
            14714 18844 0 19 mc:reader_7
            14714 18847 0 19 mc:reader_8
            14714 18848 0 19 mc:reader_9
            14714 18849 0 19 mc:reader_10
            14714 18850 0 19 mc:reader_11
            14714 18851 0 19 mc:reader_12
            14714 18852 0 19 mc:reader_13
            14714 18853 0 19 mc:reader_14
            14714 18854 0 19 mc:reader_15
            14714 18855 0 19 mc:writer_0
            14714 18856 19 0 mc:writer_1
            14714 18857 19 0 mc:writer_2
            14714 18858 19 0 mc:writer_3
            14714 18859 19 0 mc:auxIO_0
            14714 18860 19 0 mc:auxIO_1
            14714 18861 19 0 mc:auxIO_2
            14714 18862 19 0 mc:nonIO_0
            14714 18863 19 0 mc:nonIO_1
            14714 18864 19 0 mc:nonIO_2
            14714 18865 19 0 mc:nonIO_3
            14714 18866 19 0 mc:nonIO_4
            14714 18867 19 0 mc:nonIO_5
            14714 18868 19 0 mc:nonIO_6
            {code}

            Note priority (3rd column) of the auxIO and nonIO is the same as the writers (19) when it should be the same as the readers (0).

            See http://review.couchbase.org/#/c/117651/ for the patch which introduced the priority change.
            While investigating MB-36785, I observed the NonIO and AuxIO threads are incorrect set to the lowest priority - this should only be the case for Writer threads:

            {code}
            $ ps -o pid,tid,ni,pri,comm -p $(pgrep memcached|head -n1) -T|grep "reader\|writer\|aux\|non"
            14714 18837 0 19 mc:reader_0
            ...
            14714 18854 0 19 mc:reader_15
            14714 18855 0 19 mc:writer_0
            14714 18856 19 0 mc:writer_1
            14714 18857 19 0 mc:writer_2
            14714 18858 19 0 mc:writer_3
            14714 18859 19 0 mc:auxIO_0
            14714 18860 19 0 mc:auxIO_1
            14714 18861 19 0 mc:auxIO_2
            14714 18862 19 0 mc:nonIO_0
            14714 18863 19 0 mc:nonIO_1
            14714 18864 19 0 mc:nonIO_2
            14714 18865 19 0 mc:nonIO_3
            14714 18866 19 0 mc:nonIO_4
            14714 18867 19 0 mc:nonIO_5
            14714 18868 19 0 mc:nonIO_6
            {code}

            Note priority (3rd column) of the auxIO and nonIO is the same as the writers (19) when it should be the same as the readers (0).

            See http://review.couchbase.org/#/c/117651/ for the patch which introduced the priority change.
            owend Daniel Owen made changes -
            Sprint KV-Engine Mad-Hatter GA [ 910 ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            drigby Dave Rigby made changes -
            Triage Untriaged [ 10351 ] Triaged [ 10350 ]
            drigby Dave Rigby made changes -
            Labels performance
            drigby Dave Rigby made changes -
            Description While investigating MB-36785, I observed the NonIO and AuxIO threads are incorrect set to the lowest priority - this should only be the case for Writer threads:

            {code}
            $ ps -o pid,tid,ni,pri,comm -p $(pgrep memcached|head -n1) -T|grep "reader\|writer\|aux\|non"
            14714 18837 0 19 mc:reader_0
            ...
            14714 18854 0 19 mc:reader_15
            14714 18855 0 19 mc:writer_0
            14714 18856 19 0 mc:writer_1
            14714 18857 19 0 mc:writer_2
            14714 18858 19 0 mc:writer_3
            14714 18859 19 0 mc:auxIO_0
            14714 18860 19 0 mc:auxIO_1
            14714 18861 19 0 mc:auxIO_2
            14714 18862 19 0 mc:nonIO_0
            14714 18863 19 0 mc:nonIO_1
            14714 18864 19 0 mc:nonIO_2
            14714 18865 19 0 mc:nonIO_3
            14714 18866 19 0 mc:nonIO_4
            14714 18867 19 0 mc:nonIO_5
            14714 18868 19 0 mc:nonIO_6
            {code}

            Note priority (3rd column) of the auxIO and nonIO is the same as the writers (19) when it should be the same as the readers (0).

            See http://review.couchbase.org/#/c/117651/ for the patch which introduced the priority change.
            While investigating MB-36765, I observed the NonIO and AuxIO threads are incorrect set to the lowest priority - this should only be the case for Writer threads:

            {code}
            $ ps -o pid,tid,ni,pri,comm -p $(pgrep memcached|head -n1) -T|grep "reader\|writer\|aux\|non"
            14714 18837 0 19 mc:reader_0
            ...
            14714 18854 0 19 mc:reader_15
            14714 18855 0 19 mc:writer_0
            14714 18856 19 0 mc:writer_1
            14714 18857 19 0 mc:writer_2
            14714 18858 19 0 mc:writer_3
            14714 18859 19 0 mc:auxIO_0
            14714 18860 19 0 mc:auxIO_1
            14714 18861 19 0 mc:auxIO_2
            14714 18862 19 0 mc:nonIO_0
            14714 18863 19 0 mc:nonIO_1
            14714 18864 19 0 mc:nonIO_2
            14714 18865 19 0 mc:nonIO_3
            14714 18866 19 0 mc:nonIO_4
            14714 18867 19 0 mc:nonIO_5
            14714 18868 19 0 mc:nonIO_6
            {code}

            Note priority (3rd column) of the auxIO and nonIO is the same as the writers (19) when it should be the same as the readers (0).

            See http://review.couchbase.org/#/c/117651/ for the patch which introduced the priority change.
            dfinlay Dave Finlay added a comment -

            Approved as a perf regression and given the fix is simple and low risk - restoring non-I/O and aux-I/O threads to their former priority levels.

            dfinlay Dave Finlay added a comment - Approved as a perf regression and given the fix is simple and low risk - restoring non-I/O and aux-I/O threads to their former priority levels.
            dfinlay Dave Finlay made changes -
            Labels performance approved-for-mad-hatter performance
            dfinlay Dave Finlay made changes -
            Link This issue blocks MB-36676 [ MB-36676 ]
            wayne Wayne Siu made changes -
            Link This issue relates to MB-36765 [ MB-36765 ]

            Build couchbase-server-6.5.0-4923 contains kv_engine commit f4a3c7c with commit message:
            MB-37144: Don't set AuxIO & NonIO thread pri to lowest

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4923 contains kv_engine commit f4a3c7c with commit message: MB-37144 : Don't set AuxIO & NonIO thread pri to lowest
            drigby Dave Rigby made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]

            Build couchbase-server-7.0.0-1111 contains kv_engine commit f4a3c7c with commit message:
            MB-37144: Don't set AuxIO & NonIO thread pri to lowest

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1111 contains kv_engine commit f4a3c7c with commit message: MB-37144 : Don't set AuxIO & NonIO thread pri to lowest
            ritam.sharma Ritam Sharma made changes -
            Assignee Dave Rigby [ drigby ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
            wayne Wayne Siu added a comment -

            Verified the performance regression in MB-36765 is fixed in build 6.5.0-4923.
            Closing this ticket as a result.

            wayne Wayne Siu added a comment - Verified the performance regression in MB-36765 is fixed in build 6.5.0-4923. Closing this ticket as a result.
            wayne Wayne Siu made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              drigby Dave Rigby
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty