Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40371

[Doc_Isolation] xattr::utils::check_len(2617569397) exceeds 286

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Operating System:
      Centos 64-bit
    • Story Points:
      1
    • Is this a Regression?:
      No

      Description

       

      Build: 6.6.0-7861

      Scenario:

      • 4 node cluster, Couchbase bucket (replicas=2)
      • Start two transactions in parallel (One will succeed and other will rollback)
      • Rebalance out 1 node from the cluster
      • +----------------+----------------------+--------------+
        | Nodes          | Services             | Status       |
        +----------------+----------------------+--------------+
        | 172.23.105.205 | kv                   | Cluster node |
        | 172.23.105.155 | fts, index, kv, n1ql | Cluster node |
        | 172.23.105.206 | [u'kv']              | --- OUT ---> |
        | 172.23.105.159 | kv                   | Cluster node |
        +----------------+----------------------+--------------+
        

      Observation:

      During rebalance-out, rebalance operation fails with reason "mover crash - child interrupted-socket closed" message

      Failure logs:

      Rebalance exited with reason {mover_crashed,
      {unexpected_exit,
      {'EXIT',<0.15736.1>,
      {{{{{child_interrupted,
      {'EXIT',<0.3124.0>,socket_closed}},
      [{dcp_replicator,spawn_and_wait,1,
      [{file,"src/dcp_replicator.erl"}, {line,266}]},
      {dcp_replicator,handle_call,3,
      [{file,"src/dcp_replicator.erl"}, {line,121}]},
      {gen_server,try_handle_call,4,
      [{file,"gen_server.erl"},{line,636}]},
      {gen_server,handle_msg,6,
      [{file,"gen_server.erl"},{line,665}]},
      {proc_lib,init_p_do_apply,3,
      [{file,"proc_lib.erl"},{line,247}]}]},
      {gen_server,call, [<0.3123.0>,get_partitions,infinity]}},
      {gen_server,call, ['dcp_replication_manager-default',
      {get_replicator_pid,418}, infinity]}},
      {gen_server,call,
      [{'janitor_agent-default', 'ns_1@172.23.105.155'},
      {if_rebalance,<0.3785.0>,
      {update_vbucket_state,779,active,
      undefined,undefined,undefined}}, infinity]}}}}}.
      Rebalance Operation Id = 03fc94d6190f5b191aa426459e553a9c
       
      Worker <0.14842.1> (for action {move,{779,
      ['ns_1@172.23.105.206',
      'ns_1@172.23.105.155',
      'ns_1@172.23.105.159'],
      ['ns_1@172.23.105.155',
      'ns_1@172.23.105.159',
      'ns_1@172.23.105.205'],
      []}}) exited with reason {unexpected_exit,
      {'EXIT', <0.15736.1>,
      {{{{{child_interrupted,
      {'EXIT', <0.3124.0>,
      socket_closed}},
      [{dcp_replicator, spawn_and_wait, 1,
      [{file, "src/dcp_replicator.erl"}, {line, 266}]},
      {dcp_replicator, handle_call, 3,
      [{file, "src/dcp_replicator.erl"}, {line, 121}]},
      {gen_server, try_handle_call, 4,
      [{file, "gen_server.erl"}, {line, 636}]},
      {gen_server, handle_msg, 6,
      [{file, "gen_server.erl"}, {line, 665}]},
      {proc_lib, init_p_do_apply, 3,
      [{file, "proc_lib.erl"}, {line, 247}]}]},
      {gen_server, call, [<0.3123.0>,
      get_partitions, infinity]}},
      {gen_server, call,
      ['dcp_replication_manager-default',
      {get_replicator_pid, 418}, infinity]}},
      {gen_server, call,
      [{'janitor_agent-default',
      'ns_1@172.23.105.155'},
      {if_rebalance, <0.3785.0>,
      {update_vbucket_state, 779, active, undefined, undefined, undefined}},
      infinity]}}}} 

      cbcollect logs:

       
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-09T182212-ns_1%40172.23.105.155.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-09T182212-ns_1%40172.23.105.159.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-09T182212-ns_1%40172.23.105.205.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-09T182212-ns_1%40172.23.105.206.zip
      Testcase:

      ./testrunner -i /tmp/5-centos-nodes-jython.ini rerun=False,get-cbcollect-info=False -t Atomicity.doc_isolation.IsolationDocTest.test_transaction_with_rebalance,nodes_init=4,replicas=2,rebalance_type=out,nodes_out=1,doc_op=create,GROUP=P1
      

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            ashwin.govindarajulu Ashwin Govindarajulu created issue -
            Hide
            owend Daniel Owen added a comment -

            On 172.23.105.155:

            2020-07-09T11:20:11.353622-07:00 ERROR 57: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:34841 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007fa7597a7810","ewouldblock":false,"packet":{"bodylen":325,"cas":1594318811276902400,"datatype":["Snappy","Xattr"],"extlen":21,"key":"<ud>test_docs-00033425</ud>","keylen":18,"magic":"ClientRequest","opaque":430,"opcode":"DCP_DELETION","vbucket":913},"refcount":1}] - closing connection ([ 127.0.0.1:34841 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2617569397) exceeds 286
            

            and

            2020-07-09T11:20:17.268890-07:00 ERROR 57: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:35431 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007fa7591a8e10","ewouldblock":false,"packet":{"bodylen":336,"cas":1594318816006373376,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00033383</ud>","keylen":18,"magic":"ClientRequest","opaque":141,"opcode":"DCP_PREPARE","vbucket":338},"refcount":1}] - closing connection ([ 127.0.0.1:35431 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2634346613) exceeds 287
            

            Show
            owend Daniel Owen added a comment - On 172.23.105.155: 2020-07-09T11:20:11.353622-07:00 ERROR 57: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:34841 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007fa7597a7810","ewouldblock":false,"packet":{"bodylen":325,"cas":1594318811276902400,"datatype":["Snappy","Xattr"],"extlen":21,"key":"<ud>test_docs-00033425</ud>","keylen":18,"magic":"ClientRequest","opaque":430,"opcode":"DCP_DELETION","vbucket":913},"refcount":1}] - closing connection ([ 127.0.0.1:34841 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2617569397) exceeds 286 and 2020-07-09T11:20:17.268890-07:00 ERROR 57: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:35431 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007fa7591a8e10","ewouldblock":false,"packet":{"bodylen":336,"cas":1594318816006373376,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00033383</ud>","keylen":18,"magic":"ClientRequest","opaque":141,"opcode":"DCP_PREPARE","vbucket":338},"refcount":1}] - closing connection ([ 127.0.0.1:35431 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2634346613) exceeds 287
            Hide
            owend Daniel Owen added a comment -

            On 172.23.105.159:

            2020-07-09T11:06:30.597186-07:00 ERROR 66: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:40961 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f8c90437410","ewouldblock":false,"packet":{"bodylen":336,"cas":1594317990558040064,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00166742</ud>","keylen":18,"magic":"ClientRequest","opaque":34,"opcode":"DCP_PREPARE","vbucket":395},"refcount":1}] - closing connection ([ 127.0.0.1:40961 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2634346613) exceeds 287
            

            Show
            owend Daniel Owen added a comment - On 172.23.105.159: 2020-07-09T11:06:30.597186-07:00 ERROR 66: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:40961 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f8c90437410","ewouldblock":false,"packet":{"bodylen":336,"cas":1594317990558040064,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00166742</ud>","keylen":18,"magic":"ClientRequest","opaque":34,"opcode":"DCP_PREPARE","vbucket":395},"refcount":1}] - closing connection ([ 127.0.0.1:40961 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2634346613) exceeds 287
            Hide
            owend Daniel Owen added a comment -

            On 172.23.105.205:

            2020-07-09T11:20:12.513736-07:00 ERROR 78: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:44943 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f981d692610","ewouldblock":false,"packet":{"bodylen":336,"cas":1594318811468529664,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00033383</ud>","keylen":18,"magic":"ClientRequest","opaque":105,"opcode":"DCP_PREPARE","vbucket":338},"refcount":1}] - closing connection ([ 127.0.0.1:44943 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2634346613) exceeds 287
            

            Show
            owend Daniel Owen added a comment - On 172.23.105.205: 2020-07-09T11:20:12.513736-07:00 ERROR 78: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:44943 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f981d692610","ewouldblock":false,"packet":{"bodylen":336,"cas":1594318811468529664,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00033383</ud>","keylen":18,"magic":"ClientRequest","opaque":105,"opcode":"DCP_PREPARE","vbucket":338},"refcount":1}] - closing connection ([ 127.0.0.1:44943 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2634346613) exceeds 287
            Hide
            owend Daniel Owen added a comment -

            On 172.23.105.206:

            2020-07-09T11:06:30.603926-07:00 ERROR 66: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:41454 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f560c243010","ewouldblock":false,"packet":{"bodylen":336,"cas":1594317990559481856,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00166740</ud>","keylen":18,"magic":"ClientRequest","opaque":38,"opcode":"DCP_PREPARE","vbucket":901},"refcount":1}] - closing connection ([ 127.0.0.1:41454 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2634346613) exceeds 287
            

            Show
            owend Daniel Owen added a comment - On 172.23.105.206: 2020-07-09T11:06:30.603926-07:00 ERROR 66: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:41454 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f560c243010","ewouldblock":false,"packet":{"bodylen":336,"cas":1594317990559481856,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00166740</ud>","keylen":18,"magic":"ClientRequest","opaque":38,"opcode":"DCP_PREPARE","vbucket":901},"refcount":1}] - closing connection ([ 127.0.0.1:41454 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2634346613) exceeds 287
            Hide
            owend Daniel Owen added a comment -

            Duplicate of MB-40370

            Show
            owend Daniel Owen added a comment - Duplicate of MB-40370
            owend Daniel Owen made changes -
            Field Original Value New Value
            Link This issue duplicates MB-40370 [ MB-40370 ]
            owend Daniel Owen made changes -
            Summary [Doc_Isolation] Rebalance failed with reason "mover crashed - child_interrupted" [Doc_Isolation] xattr::utils::check_len(2617569397) exceeds 286
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
            owend Daniel Owen made changes -
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Resolved [ 5 ]
            Hide
            ashwin.govindarajulu Ashwin Govindarajulu added a comment -

            Closing the duplicate ticket.

            Show
            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Closing the duplicate ticket.
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

              People

              Assignee:
              ashwin.govindarajulu Ashwin Govindarajulu
              Reporter:
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty