Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40440

[Transaction] xattr::utils::check_len(2651254901) exceeds 304

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 6.6.0
    • Fix Version/s: 6.6.0
    • Component/s: couchbase-bucket
    • Environment:
      6.6.0-7873-enterprise
      couchbase-transactions-1.1.0-SNAPSHOT.jar
      java-client-3.0.6-SNAPSHOT.jar
    • Triage:
      Untriaged
    • Operating System:
      Centos 64-bit
    • Story Points:
      1
    • Is this a Regression?:
      Unknown

      Description

      Build: 6.6.0-7873

      Scenario:

      • 3 node cluster, 3 couchbase buckets

        +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+
        | Node           | Services | CPU_utilization | Mem_total  | Mem_free   | Swap_mem_used          | Active / Replica  | Version               |
        +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+
        | 172.23.98.215  | kv       | 36.0655737705   | 4339630080 | 3580170240 | 79736832 / 3758092288  | 10214 / 20462     | 6.6.0-7873-enterprise |
        | 172.23.123.111 | kv       | 18.6868686869   | 4201672704 | 3366232064 | 30670848 / 3758092288  | 10278 / 20473     | 6.6.0-7873-enterprise |
        | 172.23.123.116 | kv       | 48.7046632124   | 4201672704 | 3422486528 | 116359168 / 3758092288 | 10020 / 20118     | 6.6.0-7873-enterprise |
        +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+

      • 2020-07-13 10:48:20,966 | test  | INFO    | MainThread | [table_view:display:72] Bucket statistics
        +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
        | Bucket   | Type    | Replicas | Durability | TTL | Items | RAM Quota  | RAM Used | Disk Used |
        +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
        | bucket-0 | membase | 2        | none       | 0   | 9920  | 1541406720 | 90595104 | 470784378 |
        | bucket-1 | membase | 2        | none       | 0   | 10673 | 1541406720 | 95380544 | 443875257 |
        | bucket-2 | membase | 2        | none       | 0   | 9919  | 1541406720 | 90737664 | 472684883 |
        +----------+---------+----------+------------+-----+-------+------------+----------+-----------+ 

      • Load initial docs into all 3 buckets using transaction
      • Swap rebalance 172.23.123.111 <—> 172.23.123.115

      Observation:

      Swap rebalance at step#3 fails with reason "mover_crashed wait_seqno_persisted_failed bucket-1",

      Rebalance exited with reason {mover_crashed,
      {unexpected_exit,
      {'EXIT',<0.13721.0>,
      {{wait_seqno_persisted_failed,"bucket-1",341, 92,
      [{'ns_1@172.23.123.115',
      {'EXIT',
      {{{{{badmatch, [{<25251.5355.0>,
      {done,exit,
      {socket_closed, {gen_server,call, [<25251.5288.0>,
      {setup_streams,[339,340,341]}, infinity]}},
      [{gen_server,call,3, [{file,"gen_server.erl"}, {line,214}]},
      {dcp_replicator, '-spawn_and_wait/1-fun-0-',1, [{file, "src/dcp_replicator.erl"}, {line,243}]}]}}]},
      [{misc, sync_shutdown_many_i_am_trapping_exits, 1, [{file,"src/misc.erl"}, {line,1374}]},
      {dcp_replicator,spawn_and_wait,1, [{file,"src/dcp_replicator.erl"}, {line,265}]},
      {dcp_replicator,handle_call,3, [{file,"src/dcp_replicator.erl"}, {line,121}]},
      {gen_server,try_handle_call,4, [{file,"gen_server.erl"}, {line,636}]},
      {gen_server,handle_msg,6, [{file,"gen_server.erl"}, {line,665}]},
      {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"}, {line,247}]}]},
      {gen_server,call, [<25251.5287.0>,get_partitions, infinity]}},
      {gen_server,call,
      ['dcp_replication_manager-bucket-1',
      {get_replicator_pid,340}, infinity]}},
      {gen_server,call,
      [{'janitor_agent-bucket-1',
      'ns_1@172.23.123.115'},
      {if_rebalance,<0.13522.0>,
      {wait_seqno_persisted,341,92}}, infinity]}}}}]},
      [{ns_single_vbucket_mover, '-wait_seqno_persisted_many/5-fun-2-',5, [{file,"src/ns_single_vbucket_mover.erl"}, {line,488}]},
      {proc_lib,init_p,3, [{file,"proc_lib.erl"},{line,232}]}]}}}}.
      Rebalance Operation Id = 2c18c85d730aa4c35cdf7f72b414ddd4
       
      Worker <0.13559.0> (for action {move,{341,
      ['ns_1@172.23.123.111',
      'ns_1@172.23.98.215',
      'ns_1@172.23.123.116'],
      ['ns_1@172.23.123.115',
      'ns_1@172.23.98.215',
      'ns_1@172.23.123.116'],
      []}}) exited with reason {unexpected_exit,
      {'EXIT',
      <0.13721.0>,
      {{wait_seqno_persisted_failed,"bucket-1",341,92,
      [{'ns_1@172.23.123.115',
      {'EXIT',
      {{{{{badmatch,[{<25251.5355.0>,
      {done,exit,
      {socket_closed,
      {gen_server,call,
      [<25251.5288.0>,
      {setup_streams,
      [339,340,341]},
      infinity]}},
      [{gen_server,call,3,[{file,"gen_server.erl"},{line,214}]},
      {dcp_replicator,'-spawn_and_wait/1-fun-0-',1,[{file,"src/dcp_replicator.erl"},{line,243}]}]}}]},
      [{misc,sync_shutdown_many_i_am_trapping_exits,1,[{file,"src/misc.erl"},{line,1374}]},
      {dcp_replicator,spawn_and_wait,1,[{file,"src/dcp_replicator.erl"},{line,265}]},
      {dcp_replicator,handle_call,3,[{file,"src/dcp_replicator.erl"},{line,121}]},
      {gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,636}]},
      {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,665}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},
      {gen_server,call,[<25251.5287.0>,get_partitions,infinity]}},
      {gen_server,call,['dcp_replication_manager-bucket-1',
      {get_replicator_pid,340},infinity]}},
      {gen_server,call,
      [{'janitor_agent-bucket-1','ns_1@172.23.123.115'},
      {if_rebalance,<0.13522.0>,
      {wait_seqno_persisted,341,92}},infinity]}}}}]},
      [{ns_single_vbucket_mover,'-wait_seqno_persisted_many/5-fun-2-',5,
      [{file,"src/ns_single_vbucket_mover.erl"},{line,488}]},
      {proc_lib,init_p,3,[{file,"proc_lib.erl"},{line, 232}]}]}}}

      cbcollect logs:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.111.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.115.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.116.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.98.215.zip

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            ashwin.govindarajulu Ashwin Govindarajulu created issue -
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Field Original Value New Value
            Description *Build*: 6.6.0-7873

            *Scenario*:
             * 3 node cluster, 3 couchbase buckets
            {noformat}
            +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+
            | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
            +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+
            | 172.23.98.215 | kv | 36.0655737705 | 4339630080 | 3580170240 | 79736832 / 3758092288 | 10214 / 20462 | 6.6.0-7873-enterprise |
            | 172.23.123.111 | kv | 18.6868686869 | 4201672704 | 3366232064 | 30670848 / 3758092288 | 10278 / 20473 | 6.6.0-7873-enterprise |
            | 172.23.123.116 | kv | 48.7046632124 | 4201672704 | 3422486528 | 116359168 / 3758092288 | 10020 / 20118 | 6.6.0-7873-enterprise |
            +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+2020-07-13 10:48:20,966 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
            +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
            | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
            +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
            | bucket-0 | membase | 2 | none | 0 | 9920 | 1541406720 | 90595104 | 470784378 |
            | bucket-1 | membase | 2 | none | 0 | 10673 | 1541406720 | 95380544 | 443875257 |
            | bucket-2 | membase | 2 | none | 0 | 9919 | 1541406720 | 90737664 | 472684883 |
            +----------+---------+----------+------------+-----+-------+------------+----------+-----------+ {noformat}

             * Load initial docs into all 3 buckets using transaction
             * Swap rebalance 172.23.123.111 <—> 172.23.123.115

            *Observation*:

            Swap rebalance at step#3 fails with reason "_*mover_crashed wait_seqno_persisted_failed bucket-1*_",
            {noformat}
            Rebalance exited with reason {mover_crashed,
            {unexpected_exit,
            {'EXIT',<0.13721.0>,
            {{wait_seqno_persisted_failed,"bucket-1",341, 92,
            [{'ns_1@172.23.123.115',
            {'EXIT',
            {{{{{badmatch, [{<25251.5355.0>,
            {done,exit,
            {socket_closed, {gen_server,call, [<25251.5288.0>,
            {setup_streams,[339,340,341]}, infinity]}},
            [{gen_server,call,3, [{file,"gen_server.erl"}, {line,214}]},
            {dcp_replicator, '-spawn_and_wait/1-fun-0-',1, [{file, "src/dcp_replicator.erl"}, {line,243}]}]}}]},
            [{misc, sync_shutdown_many_i_am_trapping_exits, 1, [{file,"src/misc.erl"}, {line,1374}]},
            {dcp_replicator,spawn_and_wait,1, [{file,"src/dcp_replicator.erl"}, {line,265}]},
            {dcp_replicator,handle_call,3, [{file,"src/dcp_replicator.erl"}, {line,121}]},
            {gen_server,try_handle_call,4, [{file,"gen_server.erl"}, {line,636}]},
            {gen_server,handle_msg,6, [{file,"gen_server.erl"}, {line,665}]},
            {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"}, {line,247}]}]},
            {gen_server,call, [<25251.5287.0>,get_partitions, infinity]}},
            {gen_server,call,
            ['dcp_replication_manager-bucket-1',
            {get_replicator_pid,340}, infinity]}},
            {gen_server,call,
            [{'janitor_agent-bucket-1',
            'ns_1@172.23.123.115'},
            {if_rebalance,<0.13522.0>,
            {wait_seqno_persisted,341,92}}, infinity]}}}}]},
            [{ns_single_vbucket_mover, '-wait_seqno_persisted_many/5-fun-2-',5, [{file,"src/ns_single_vbucket_mover.erl"}, {line,488}]},
            {proc_lib,init_p,3, [{file,"proc_lib.erl"},{line,232}]}]}}}}.
            Rebalance Operation Id = 2c18c85d730aa4c35cdf7f72b414ddd4

            Worker <0.13559.0> (for action {move,{341,
            ['ns_1@172.23.123.111',
            'ns_1@172.23.98.215',
            'ns_1@172.23.123.116'],
            ['ns_1@172.23.123.115',
            'ns_1@172.23.98.215',
            'ns_1@172.23.123.116'],
            []}}) exited with reason {unexpected_exit,
            {'EXIT',
            <0.13721.0>,
            {{wait_seqno_persisted_failed,"bucket-1",341,92,
            [{'ns_1@172.23.123.115',
            {'EXIT',
            {{{{{badmatch,[{<25251.5355.0>,
            {done,exit,
            {socket_closed,
            {gen_server,call,
            [<25251.5288.0>,
            {setup_streams,
            [339,340,341]},
            infinity]}},
            [{gen_server,call,3,[{file,"gen_server.erl"},{line,214}]},
            {dcp_replicator,'-spawn_and_wait/1-fun-0-',1,[{file,"src/dcp_replicator.erl"},{line,243}]}]}}]},
            [{misc,sync_shutdown_many_i_am_trapping_exits,1,[{file,"src/misc.erl"},{line,1374}]},
            {dcp_replicator,spawn_and_wait,1,[{file,"src/dcp_replicator.erl"},{line,265}]},
            {dcp_replicator,handle_call,3,[{file,"src/dcp_replicator.erl"},{line,121}]},
            {gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,636}]},
            {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,665}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},
            {gen_server,call,[<25251.5287.0>,get_partitions,infinity]}},
            {gen_server,call,['dcp_replication_manager-bucket-1',
            {get_replicator_pid,340},infinity]}},
            {gen_server,call,
            [{'janitor_agent-bucket-1','ns_1@172.23.123.115'},
            {if_rebalance,<0.13522.0>,
            {wait_seqno_persisted,341,92}},infinity]}}}}]},
            [{ns_single_vbucket_mover,'-wait_seqno_persisted_many/5-fun-2-',5,
            [{file,"src/ns_single_vbucket_mover.erl"},{line,488}]},
            {proc_lib,init_p,3,[{file,"proc_lib.erl"},{line, 232}]}]}}}{noformat}
            *cbcollect logs:*

            [https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.111.zip]
            [https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.115.zip]
            [https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.116.zip]
            [https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.98.215.zip]
            *Build*: 6.6.0-7873

            *Scenario*:
             * 3 node cluster, 3 couchbase buckets
            {noformat}+----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+
            | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
            +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+
            | 172.23.98.215 | kv | 36.0655737705 | 4339630080 | 3580170240 | 79736832 / 3758092288 | 10214 / 20462 | 6.6.0-7873-enterprise |
            | 172.23.123.111 | kv | 18.6868686869 | 4201672704 | 3366232064 | 30670848 / 3758092288 | 10278 / 20473 | 6.6.0-7873-enterprise |
            | 172.23.123.116 | kv | 48.7046632124 | 4201672704 | 3422486528 | 116359168 / 3758092288 | 10020 / 20118 | 6.6.0-7873-enterprise |
            +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+{noformat}
             *
            {noformat}2020-07-13 10:48:20,966 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
            +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
            | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
            +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
            | bucket-0 | membase | 2 | none | 0 | 9920 | 1541406720 | 90595104 | 470784378 |
            | bucket-1 | membase | 2 | none | 0 | 10673 | 1541406720 | 95380544 | 443875257 |
            | bucket-2 | membase | 2 | none | 0 | 9919 | 1541406720 | 90737664 | 472684883 |
            +----------+---------+----------+------------+-----+-------+------------+----------+-----------+ {noformat}

             * Load initial docs into all 3 buckets using transaction
             * Swap rebalance 172.23.123.111 <—> 172.23.123.115

            *Observation*:

            Swap rebalance at step#3 fails with reason "_*mover_crashed wait_seqno_persisted_failed bucket-1*_",
            {noformat}Rebalance exited with reason {mover_crashed,
            {unexpected_exit,
            {'EXIT',<0.13721.0>,
            {{wait_seqno_persisted_failed,"bucket-1",341, 92,
            [{'ns_1@172.23.123.115',
            {'EXIT',
            {{{{{badmatch, [{<25251.5355.0>,
            {done,exit,
            {socket_closed, {gen_server,call, [<25251.5288.0>,
            {setup_streams,[339,340,341]}, infinity]}},
            [{gen_server,call,3, [{file,"gen_server.erl"}, {line,214}]},
            {dcp_replicator, '-spawn_and_wait/1-fun-0-',1, [{file, "src/dcp_replicator.erl"}, {line,243}]}]}}]},
            [{misc, sync_shutdown_many_i_am_trapping_exits, 1, [{file,"src/misc.erl"}, {line,1374}]},
            {dcp_replicator,spawn_and_wait,1, [{file,"src/dcp_replicator.erl"}, {line,265}]},
            {dcp_replicator,handle_call,3, [{file,"src/dcp_replicator.erl"}, {line,121}]},
            {gen_server,try_handle_call,4, [{file,"gen_server.erl"}, {line,636}]},
            {gen_server,handle_msg,6, [{file,"gen_server.erl"}, {line,665}]},
            {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"}, {line,247}]}]},
            {gen_server,call, [<25251.5287.0>,get_partitions, infinity]}},
            {gen_server,call,
            ['dcp_replication_manager-bucket-1',
            {get_replicator_pid,340}, infinity]}},
            {gen_server,call,
            [{'janitor_agent-bucket-1',
            'ns_1@172.23.123.115'},
            {if_rebalance,<0.13522.0>,
            {wait_seqno_persisted,341,92}}, infinity]}}}}]},
            [{ns_single_vbucket_mover, '-wait_seqno_persisted_many/5-fun-2-',5, [{file,"src/ns_single_vbucket_mover.erl"}, {line,488}]},
            {proc_lib,init_p,3, [{file,"proc_lib.erl"},{line,232}]}]}}}}.
            Rebalance Operation Id = 2c18c85d730aa4c35cdf7f72b414ddd4

            Worker <0.13559.0> (for action {move,{341,
            ['ns_1@172.23.123.111',
            'ns_1@172.23.98.215',
            'ns_1@172.23.123.116'],
            ['ns_1@172.23.123.115',
            'ns_1@172.23.98.215',
            'ns_1@172.23.123.116'],
            []}}) exited with reason {unexpected_exit,
            {'EXIT',
            <0.13721.0>,
            {{wait_seqno_persisted_failed,"bucket-1",341,92,
            [{'ns_1@172.23.123.115',
            {'EXIT',
            {{{{{badmatch,[{<25251.5355.0>,
            {done,exit,
            {socket_closed,
            {gen_server,call,
            [<25251.5288.0>,
            {setup_streams,
            [339,340,341]},
            infinity]}},
            [{gen_server,call,3,[{file,"gen_server.erl"},{line,214}]},
            {dcp_replicator,'-spawn_and_wait/1-fun-0-',1,[{file,"src/dcp_replicator.erl"},{line,243}]}]}}]},
            [{misc,sync_shutdown_many_i_am_trapping_exits,1,[{file,"src/misc.erl"},{line,1374}]},
            {dcp_replicator,spawn_and_wait,1,[{file,"src/dcp_replicator.erl"},{line,265}]},
            {dcp_replicator,handle_call,3,[{file,"src/dcp_replicator.erl"},{line,121}]},
            {gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,636}]},
            {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,665}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},
            {gen_server,call,[<25251.5287.0>,get_partitions,infinity]}},
            {gen_server,call,['dcp_replication_manager-bucket-1',
            {get_replicator_pid,340},infinity]}},
            {gen_server,call,
            [{'janitor_agent-bucket-1','ns_1@172.23.123.115'},
            {if_rebalance,<0.13522.0>,
            {wait_seqno_persisted,341,92}},infinity]}}}}]},
            [{ns_single_vbucket_mover,'-wait_seqno_persisted_many/5-fun-2-',5,
            [{file,"src/ns_single_vbucket_mover.erl"},{line,488}]},
            {proc_lib,init_p,3,[{file,"proc_lib.erl"},{line, 232}]}]}}}{noformat}
            *cbcollect logs:*

            [https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.111.zip]
             [https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.115.zip]
             [https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.116.zip]
             [https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.98.215.zip]
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Attachment test.log [ 100776 ]
            Hide
            owend Daniel Owen added a comment -

            On 172.23.123.115:

            2020-07-13T10:48:28.454765-07:00 ERROR 81: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:38511 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007fd724149a10","ewouldblock":false,"packet":{"bodylen":356,"cas":1594662508245549056,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013799</ud>","keylen":18,"magic":"ClientRequest","opaque":15,"opcode":"DCP_PREPARE","vbucket":341},"refcount":1}] - closing connection ([ 127.0.0.1:38511 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2668032117) exceeds 307
            

            Show
            owend Daniel Owen added a comment - On 172.23.123.115: 2020-07-13T10:48:28.454765-07:00 ERROR 81: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:38511 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007fd724149a10","ewouldblock":false,"packet":{"bodylen":356,"cas":1594662508245549056,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013799</ud>","keylen":18,"magic":"ClientRequest","opaque":15,"opcode":"DCP_PREPARE","vbucket":341},"refcount":1}] - closing connection ([ 127.0.0.1:38511 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2668032117) exceeds 307
            Hide
            owend Daniel Owen added a comment -

            On 172.23.123.116:

            2020-07-13T10:48:29.587410-07:00 ERROR 68: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:45135 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f3bcb70e910","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662509410975744,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:45135 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304
            2020-07-13T10:48:34.477007-07:00 ERROR 68: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:42260 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f3bcb70e910","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662513939513344,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:42260 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304
            2020-07-13T10:48:39.472333-07:00 ERROR 68: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:33317 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f3bcb70d510","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662518461300736,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:33317 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304
            2020-07-13T10:48:44.473541-07:00 ERROR 68: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:38133 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f3bca2dcc10","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662522984529920,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:38133 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304
            2020-07-13T10:48:50.652305-07:00 ERROR 56: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:45237 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f3c28450810","ewouldblock":false,"packet":{"bodylen":352,"cas":1594662509131071488,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00015042</ud>","keylen":18,"magic":"ClientRequest","opaque":101,"opcode":"DCP_PREPARE","vbucket":781},"refcount":1}] - closing connection ([ 127.0.0.1:45237 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 303
            

            Show
            owend Daniel Owen added a comment - On 172.23.123.116: 2020-07-13T10:48:29.587410-07:00 ERROR 68: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:45135 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f3bcb70e910","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662509410975744,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:45135 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304 2020-07-13T10:48:34.477007-07:00 ERROR 68: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:42260 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f3bcb70e910","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662513939513344,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:42260 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304 2020-07-13T10:48:39.472333-07:00 ERROR 68: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:33317 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f3bcb70d510","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662518461300736,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:33317 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304 2020-07-13T10:48:44.473541-07:00 ERROR 68: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:38133 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f3bca2dcc10","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662522984529920,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:38133 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304 2020-07-13T10:48:50.652305-07:00 ERROR 56: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:45237 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f3c28450810","ewouldblock":false,"packet":{"bodylen":352,"cas":1594662509131071488,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00015042</ud>","keylen":18,"magic":"ClientRequest","opaque":101,"opcode":"DCP_PREPARE","vbucket":781},"refcount":1}] - closing connection ([ 127.0.0.1:45237 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 303
            Hide
            owend Daniel Owen added a comment -

            On 172.23.98.215:

            2020-07-13T10:48:29.601506-07:00 ERROR 72: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:59646 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f43b5414c10","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662509410975744,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:59646 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304
            2020-07-13T10:48:34.487231-07:00 ERROR 72: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:35187 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f43b76a8f10","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662513939513344,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:35187 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304
            2020-07-13T10:48:39.496428-07:00 ERROR 72: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:54045 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f43b76a8f10","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662518461300736,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:54045 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304
            2020-07-13T10:48:44.487551-07:00 ERROR 72: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:36078 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f43b76a8f10","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662522984529920,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:36078 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304
            2020-07-13T10:48:50.703677-07:00 ERROR 51: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:46115 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f43c563ac10","ewouldblock":false,"packet":{"bodylen":351,"cas":1594662509140639744,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00016290</ud>","keylen":18,"magic":"ClientRequest","opaque":73,"opcode":"DCP_PREPARE","vbucket":412},"refcount":1}] - closing connection ([ 127.0.0.1:46115 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2634477685) exceeds 302
            

            Show
            owend Daniel Owen added a comment - On 172.23.98.215: 2020-07-13T10:48:29.601506-07:00 ERROR 72: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:59646 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f43b5414c10","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662509410975744,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:59646 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304 2020-07-13T10:48:34.487231-07:00 ERROR 72: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:35187 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f43b76a8f10","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662513939513344,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:35187 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304 2020-07-13T10:48:39.496428-07:00 ERROR 72: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:54045 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f43b76a8f10","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662518461300736,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:54045 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304 2020-07-13T10:48:44.487551-07:00 ERROR 72: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:36078 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f43b76a8f10","ewouldblock":false,"packet":{"bodylen":353,"cas":1594662522984529920,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00013807</ud>","keylen":18,"magic":"ClientRequest","opaque":118,"opcode":"DCP_PREPARE","vbucket":115},"refcount":1}] - closing connection ([ 127.0.0.1:36078 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2651254901) exceeds 304 2020-07-13T10:48:50.703677-07:00 ERROR 51: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 127.0.0.1:46115 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f43c563ac10","ewouldblock":false,"packet":{"bodylen":351,"cas":1594662509140639744,"datatype":["Snappy","Xattr"],"extlen":31,"key":"<ud>test_docs-00016290</ud>","keylen":18,"magic":"ClientRequest","opaque":73,"opcode":"DCP_PREPARE","vbucket":412},"refcount":1}] - closing connection ([ 127.0.0.1:46115 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]): xattr::utils::check_len(2634477685) exceeds 302
            Hide
            owend Daniel Owen added a comment -

            Fix for MB-40370 went into couchbase-server-6.6.0-7879.
            This MB is raised against 6.6.0-7873. Therefore marking as a duplicate.

            Show
            owend Daniel Owen added a comment - Fix for MB-40370 went into couchbase-server-6.6.0-7879. This MB is raised against 6.6.0-7873. Therefore marking as a duplicate.
            owend Daniel Owen made changes -
            Summary [Transaction] Swap rebalance failed with reason "wait_seqno_persisted_failed" [Transaction] xattr::utils::check_len(2651254901) exceeds 304
            owend Daniel Owen made changes -
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Resolved [ 5 ]
            owend Daniel Owen made changes -
            Link This issue duplicates MB-40370 [ MB-40370 ]
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
            ritam.sharma Ritam Sharma made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

              People

              Assignee:
              ashwin.govindarajulu Ashwin Govindarajulu
              Reporter:
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty