Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19640

swap rebalance failed while querying views in parallel

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 4.5.0
    • 4.5.0
    • view-engine
    • None
    • Untriaged
    • Unknown

    Description

      The following testcase failed in 4.5.0-2550:

      ./testrunner -i INI_FILE get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,demand_encryption=1,fail_on_errors=1,GROUP=ALL -t xdcr.rebalanceXDCR.Rebalance.swap_rebalance_replication_with_ddoc_compaction,items=100000,rdirection=bidirection,is_dev_ddoc=false,rebalance=C2,GROUP=P1,poll_timeout=900

      with following error:

      ======================================================================
      ERROR: swap_rebalance_replication_with_ddoc_compaction (xdcr.rebalanceXDCR.Rebalance)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
      File "pytests/xdcr/rebalanceXDCR.py", line 158, in swap_rebalance_replication_with_ddoc_compaction
      task.result(self._poll_timeout)
      File "lib/tasks/future.py", line 160, in result
      return self.__get_result()
      File "lib/tasks/future.py", line 112, in __get_result
      raise self._exception
      RebalanceFailedException: Rebalance Failed: seems like rebalance hangs. please check logs!

      ----------------------------------------------------------------------

      Error occurred when the test was trying to do a swap rebalance while querying views in parallel - attaching logs and console output

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            View compaction is stuck on .102:

                 {<11579.6188.0>,
                  [{registered_name,[]},
                   {status,waiting},
                   {initial_call,{erlang,apply,2}},
                   {backtrace,
                       [<<"Program counter: 0x00007f1b1737b7a0 (couch_set_view_compactor:compact_group/6 + 560)">>,
                        <<"CP: 0x0000000000000000 (invalid)">>,<<"arity = 0">>,<<>>,
                        <<"0x00007f1b4513b278 Return addr 0x0000000000891848 (<terminate process normally>)">>,
                        <<"y(0)     []">>,<<"y(1)     []">>,<<"y(2)     []">>,
                        <<"y(3)     []">>,<<"y(4)     []">>,
                        <<"y(5)     {1463,364536,485580}">>,
                        <<"y(6)     #Ref<0.0.0.172159>">>,
                        <<"y(7)     #Ref<0.0.0.172158>">>,
                        <<"y(8)     [{original_target,{[{type,bucket}]}},{trigger_type,manual}]">>,
                        <<"y(9)     <0.284.0>">>,
                        <<"y(10)    \"/opt/couchbase/var/lib/couchbase/data/@indexes/default/tmp_232bbf5d6dcc75f0e68ee8f79a976bcd_replica\"">>,
                        <<"(11)    {set_view_group,<<16 bytes>>,<0.6184.0>,<<7 bytes>>,<<13 bytes>>,[],[{set_view,0,<<40 bytes>>,#Ref<0.0.0.172156>">>,
                        <<"y(12)    replica">>,<<"y(13)    <<7 bytes>>">>,<<>>]},
                   {error_handler,error_handler},
                   {garbage_collection,
                       [{min_bin_vheap_size,46422},
                        {min_heap_size,233},
                        {fullsweep_after,512},
                        {minor_gcs,0}]},
                   {heap_size,28690},
                   {total_heap_size,28690},
                   {links,[<11579.284.0>]},
                   {monitors,[{process,<11579.2369.0>}]},
                   {monitored_by,[<0.30367.14>]},
                   {memory,230448},
                   {message_queue_len,0},
                   {reductions,8},
                   {trap_exit,false},
                   {current_location,
                       {couch_set_view_compactor,compact_group,6,
                           [{file,
                                "/home/couchbase/jenkins/workspace/watson-unix/couchdb/src/couch_set_view/src/couch_set_view_compactor.erl"},
                            {line,91}]}},
                   {dictionary,[]}]}
            

            Aliaksey Artamonau Aliaksey Artamonau (Inactive) added a comment - View compaction is stuck on .102: {<11579.6188.0>, [{registered_name,[]}, {status,waiting}, {initial_call,{erlang,apply,2}}, {backtrace, [<<"Program counter: 0x00007f1b1737b7a0 (couch_set_view_compactor:compact_group/6 + 560)">>, <<"CP: 0x0000000000000000 (invalid)">>,<<"arity = 0">>,<<>>, <<"0x00007f1b4513b278 Return addr 0x0000000000891848 (<terminate process normally>)">>, <<"y(0) []">>,<<"y(1) []">>,<<"y(2) []">>, <<"y(3) []">>,<<"y(4) []">>, <<"y(5) {1463,364536,485580}">>, <<"y(6) #Ref<0.0.0.172159>">>, <<"y(7) #Ref<0.0.0.172158>">>, <<"y(8) [{original_target,{[{type,bucket}]}},{trigger_type,manual}]">>, <<"y(9) <0.284.0>">>, <<"y(10) \"/opt/couchbase/var/lib/couchbase/data/@indexes/default/tmp_232bbf5d6dcc75f0e68ee8f79a976bcd_replica\"">>, <<"(11) {set_view_group,<<16 bytes>>,<0.6184.0>,<<7 bytes>>,<<13 bytes>>,[],[{set_view,0,<<40 bytes>>,#Ref<0.0.0.172156>">>, <<"y(12) replica">>,<<"y(13) <<7 bytes>>">>,<<>>]}, {error_handler,error_handler}, {garbage_collection, [{min_bin_vheap_size,46422}, {min_heap_size,233}, {fullsweep_after,512}, {minor_gcs,0}]}, {heap_size,28690}, {total_heap_size,28690}, {links,[<11579.284.0>]}, {monitors,[{process,<11579.2369.0>}]}, {monitored_by,[<0.30367.14>]}, {memory,230448}, {message_queue_len,0}, {reductions,8}, {trap_exit,false}, {current_location, {couch_set_view_compactor,compact_group,6, [{file, "/home/couchbase/jenkins/workspace/watson-unix/couchdb/src/couch_set_view/src/couch_set_view_compactor.erl"}, {line,91}]}}, {dictionary,[]}]}

            The indexing seems to still progress. So my guess it's hitting MB-19503. Please re-run the test once a fix for that issue is merged.

            vmx Volker Mische added a comment - The indexing seems to still progress. So my guess it's hitting MB-19503 . Please re-run the test once a fix for that issue is merged.
            ritam.sharma Ritam Sharma added a comment -

            MB-19503 is fixed now, bug will be updated tomorrow when tests run against build containing the fix.

            ritam.sharma Ritam Sharma added a comment - MB-19503 is fixed now, bug will be updated tomorrow when tests run against build containing the fix.

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty