Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40440

[Transaction] xattr::utils::check_len(2651254901) exceeds 304

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 6.6.0
    • Fix Version/s: 6.6.0
    • Component/s: couchbase-bucket
    • Environment:
      6.6.0-7873-enterprise
      couchbase-transactions-1.1.0-SNAPSHOT.jar
      java-client-3.0.6-SNAPSHOT.jar
    • Triage:
      Untriaged
    • Operating System:
      Centos 64-bit
    • Story Points:
      1
    • Is this a Regression?:
      Unknown

      Description

      Build: 6.6.0-7873

      Scenario:

      • 3 node cluster, 3 couchbase buckets

        +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+
        | Node           | Services | CPU_utilization | Mem_total  | Mem_free   | Swap_mem_used          | Active / Replica  | Version               |
        +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+
        | 172.23.98.215  | kv       | 36.0655737705   | 4339630080 | 3580170240 | 79736832 / 3758092288  | 10214 / 20462     | 6.6.0-7873-enterprise |
        | 172.23.123.111 | kv       | 18.6868686869   | 4201672704 | 3366232064 | 30670848 / 3758092288  | 10278 / 20473     | 6.6.0-7873-enterprise |
        | 172.23.123.116 | kv       | 48.7046632124   | 4201672704 | 3422486528 | 116359168 / 3758092288 | 10020 / 20118     | 6.6.0-7873-enterprise |
        +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+

      • 2020-07-13 10:48:20,966 | test  | INFO    | MainThread | [table_view:display:72] Bucket statistics
        +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
        | Bucket   | Type    | Replicas | Durability | TTL | Items | RAM Quota  | RAM Used | Disk Used |
        +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
        | bucket-0 | membase | 2        | none       | 0   | 9920  | 1541406720 | 90595104 | 470784378 |
        | bucket-1 | membase | 2        | none       | 0   | 10673 | 1541406720 | 95380544 | 443875257 |
        | bucket-2 | membase | 2        | none       | 0   | 9919  | 1541406720 | 90737664 | 472684883 |
        +----------+---------+----------+------------+-----+-------+------------+----------+-----------+ 

      • Load initial docs into all 3 buckets using transaction
      • Swap rebalance 172.23.123.111 <—> 172.23.123.115

      Observation:

      Swap rebalance at step#3 fails with reason "mover_crashed wait_seqno_persisted_failed bucket-1",

      Rebalance exited with reason {mover_crashed,
      {unexpected_exit,
      {'EXIT',<0.13721.0>,
      {{wait_seqno_persisted_failed,"bucket-1",341, 92,
      [{'ns_1@172.23.123.115',
      {'EXIT',
      {{{{{badmatch, [{<25251.5355.0>,
      {done,exit,
      {socket_closed, {gen_server,call, [<25251.5288.0>,
      {setup_streams,[339,340,341]}, infinity]}},
      [{gen_server,call,3, [{file,"gen_server.erl"}, {line,214}]},
      {dcp_replicator, '-spawn_and_wait/1-fun-0-',1, [{file, "src/dcp_replicator.erl"}, {line,243}]}]}}]},
      [{misc, sync_shutdown_many_i_am_trapping_exits, 1, [{file,"src/misc.erl"}, {line,1374}]},
      {dcp_replicator,spawn_and_wait,1, [{file,"src/dcp_replicator.erl"}, {line,265}]},
      {dcp_replicator,handle_call,3, [{file,"src/dcp_replicator.erl"}, {line,121}]},
      {gen_server,try_handle_call,4, [{file,"gen_server.erl"}, {line,636}]},
      {gen_server,handle_msg,6, [{file,"gen_server.erl"}, {line,665}]},
      {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"}, {line,247}]}]},
      {gen_server,call, [<25251.5287.0>,get_partitions, infinity]}},
      {gen_server,call,
      ['dcp_replication_manager-bucket-1',
      {get_replicator_pid,340}, infinity]}},
      {gen_server,call,
      [{'janitor_agent-bucket-1',
      'ns_1@172.23.123.115'},
      {if_rebalance,<0.13522.0>,
      {wait_seqno_persisted,341,92}}, infinity]}}}}]},
      [{ns_single_vbucket_mover, '-wait_seqno_persisted_many/5-fun-2-',5, [{file,"src/ns_single_vbucket_mover.erl"}, {line,488}]},
      {proc_lib,init_p,3, [{file,"proc_lib.erl"},{line,232}]}]}}}}.
      Rebalance Operation Id = 2c18c85d730aa4c35cdf7f72b414ddd4
       
      Worker <0.13559.0> (for action {move,{341,
      ['ns_1@172.23.123.111',
      'ns_1@172.23.98.215',
      'ns_1@172.23.123.116'],
      ['ns_1@172.23.123.115',
      'ns_1@172.23.98.215',
      'ns_1@172.23.123.116'],
      []}}) exited with reason {unexpected_exit,
      {'EXIT',
      <0.13721.0>,
      {{wait_seqno_persisted_failed,"bucket-1",341,92,
      [{'ns_1@172.23.123.115',
      {'EXIT',
      {{{{{badmatch,[{<25251.5355.0>,
      {done,exit,
      {socket_closed,
      {gen_server,call,
      [<25251.5288.0>,
      {setup_streams,
      [339,340,341]},
      infinity]}},
      [{gen_server,call,3,[{file,"gen_server.erl"},{line,214}]},
      {dcp_replicator,'-spawn_and_wait/1-fun-0-',1,[{file,"src/dcp_replicator.erl"},{line,243}]}]}}]},
      [{misc,sync_shutdown_many_i_am_trapping_exits,1,[{file,"src/misc.erl"},{line,1374}]},
      {dcp_replicator,spawn_and_wait,1,[{file,"src/dcp_replicator.erl"},{line,265}]},
      {dcp_replicator,handle_call,3,[{file,"src/dcp_replicator.erl"},{line,121}]},
      {gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,636}]},
      {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,665}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},
      {gen_server,call,[<25251.5287.0>,get_partitions,infinity]}},
      {gen_server,call,['dcp_replication_manager-bucket-1',
      {get_replicator_pid,340},infinity]}},
      {gen_server,call,
      [{'janitor_agent-bucket-1','ns_1@172.23.123.115'},
      {if_rebalance,<0.13522.0>,
      {wait_seqno_persisted,341,92}},infinity]}}}}]},
      [{ns_single_vbucket_mover,'-wait_seqno_persisted_many/5-fun-2-',5,
      [{file,"src/ns_single_vbucket_mover.erl"},{line,488}]},
      {proc_lib,init_p,3,[{file,"proc_lib.erl"},{line, 232}]}]}}}

      cbcollect logs:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.111.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.115.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.116.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.98.215.zip

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            There are no commits yet on this issue.

              People

              Assignee:
              ashwin.govindarajulu Ashwin Govindarajulu
              Reporter:
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty