Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40579

[System Test]: Rebalance failed with error "Closing connection: xattr::utils::check_len(1592262656) exceeds 80"

    XMLWordPrintable

    Details

      Description

      Build : 7.0.0-2635
      Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Iteration : 1st

      All rebalance operations in the longevity system test are failing with the following errors -

      [ns_server:error,2020-07-21T13:52:53.934-07:00,ns_1@172.23.97.74:<0.22710.290>:ns_single_vbucket_mover:spawn_and_wait:82]Got unexpected exit signal {'EXIT',<0.20904.290>,
                                  {{bulk_set_vbucket_state_failed,
                                    [{'ns_1@172.23.120.81',
                                      {'EXIT',
                                       {{{{{badmatch,
                                            [{<26100.16216.43>,
                                              {done,exit,
                                               {socket_closed,
                                                {gen_server,call,
                                                 [<26100.16335.43>,
                                                  {setup_streams,[1023]},
                                                  infinity]}},
                                               [{gen_server,call,3,
                                                 [{file,"gen_server.erl"},
                                                  {line,214}]},
                                                {dcp_replicator,
                                                 '-spawn_and_wait/1-fun-0-',1,
                                                 [{file,"src/dcp_replicator.erl"},
                                                  {line,243}]}]}}]},
                                           [{misc,
                                             sync_shutdown_many_i_am_trapping_exits,
                                             1,
                                             [{file,"src/misc.erl"},{line,1374}]},
                                            {dcp_replicator,spawn_and_wait,1,
                                             [{file,"src/dcp_replicator.erl"},
                                              {line,265}]},
                                            {dcp_replicator,handle_call,3,
                                             [{file,"src/dcp_replicator.erl"},
                                              {line,121}]},
                                            {gen_server,try_handle_call,4,
                                             [{file,"gen_server.erl"},{line,636}]},
                                            {gen_server,handle_msg,6,
                                             [{file,"gen_server.erl"},{line,665}]},
                                            {proc_lib,init_p_do_apply,3,
                                             [{file,"proc_lib.erl"},{line,247}]}]},
                                          {gen_server,call,
                                           [<26100.16101.43>,
                                            {setup_replication,[1023]},
                                            infinity]}},
                                         {gen_server,call,
                                          ['replication_manager-WAREHOUSE',
                                           {change_vbucket_replication,1023,
                                            'ns_1@172.23.97.74'},
                                           infinity]}},
                                        {gen_server,call,
                                          [{'janitor_agent-WAREHOUSE',
                                           'ns_1@172.23.120.81'},
                                          {if_rebalance,<0.22526.290>,
                                           {update_vbucket_state,913,replica,
                                            undefined,'ns_1@172.23.96.14'}},
                                          infinity]}}}}]},
                                   [{janitor_agent,bulk_set_vbucket_state,4,
                                     [{file,"src/janitor_agent.erl"},{line,403}]},
                                    {proc_lib,init_p,3,
                                     [{file,"proc_lib.erl"},{line,232}]}]}}
      [ns_server:error,2020-07-21T13:52:53.934-07:00,ns_1@172.23.97.74:<0.21570.290>:ns_single_vbucket_mover:spawn_and_wait:82]Got unexpected exit signal {'EXIT',<0.22588.290>,
                                  {{bulk_set_vbucket_state_failed,
                                    [{'ns_1@172.23.120.81',
                                      {'EXIT',
                                       {{{{{badmatch,
                                            [{<26100.16216.43>,
                                              {done,exit,
                                               {socket_closed,
                                                {gen_server,call,
                                                 [<26100.16335.43>,
                                                  {setup_streams,[1023]},
                                                  infinity]}},
                                               [{gen_server,call,3,
                                                 [{file,"gen_server.erl"},
                                                  {line,214}]},
                                                {dcp_replicator,
                                                 '-spawn_and_wait/1-fun-0-',1,
                                                 [{file,"src/dcp_replicator.erl"},
                                                  {line,243}]}]}}]},
                                           [{misc,
                                             sync_shutdown_many_i_am_trapping_exits,
                                             1,
                                             [{file,"src/misc.erl"},{line,1374}]},
                                            {dcp_replicator,spawn_and_wait,1,
                                             [{file,"src/dcp_replicator.erl"},
                                              {line,265}]},
                                            {dcp_replicator,handle_call,3,
                                             [{file,"src/dcp_replicator.erl"},
                                              {line,121}]},
                                            {gen_server,try_handle_call,4,
                                             [{file,"gen_server.erl"},{line,636}]},
                                            {gen_server,handle_msg,6,
                                             [{file,"gen_server.erl"},{line,665}]},
                                            {proc_lib,init_p_do_apply,3,
                                             [{file,"proc_lib.erl"},{line,247}]}]},
                                          {gen_server,call,
                                           [<26100.16101.43>,
                                            {setup_replication,[1023]},
      ...
      ...
      ...
      

      On 172.23.120.81, at the same time the following msgs can be seen in the memcached logs -

      2020-07-21T13:52:52.200735-07:00 INFO 67: Client 172.23.104.254:56086 authenticated as <ud>Administrator</ud>
      2020-07-21T13:52:52.237301-07:00 INFO 67: HELO [{"a":"couchbase-java-client/${project.version} (git: ${git.commit.id.describe}, core: ${git.commit.id.describe}) (Linux/3.10.0-1062.9.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_252-b09)","i":"323CC12376D83AC6/00000000116FA20A"}] TCP nodelay, Mutation seqno, XATTR, XERROR, Select bucket [ 172.23.104.254:46662 - 172.23.120.81:11210 (not authenticated) ]
      2020-07-21T13:52:52.256367-07:00 INFO 67: Client 172.23.104.254:46662 authenticated as <ud>Administrator</ud>
      2020-07-21T13:52:52.382994-07:00 INFO 67: Client 127.0.0.1:47011 authenticated as <ud>@ns_server</ud>
      2020-07-21T13:52:52.383340-07:00 INFO 67: HELO [regular] [ 127.0.0.1:47011 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]
      2020-07-21T13:52:52.639734-07:00 INFO (WAREHOUSE) EventuallyPersistentEngine::enableTraffic() result true
      2020-07-21T13:52:52.639752-07:00 INFO (WAREHOUSE) EventuallyPersistentEngine::handleTrafficControlCmd() Data traffic to persistence engine is enabled
      2020-07-21T13:52:52.954954-07:00 INFO (WAREHOUSE) VBucket: created vb:1023 with state:pending initialState:dead lastSeqno:0 persistedRange:{0,0} max_cas:0 uuid:162031797748993 topology:null
      2020-07-21T13:52:52.984729-07:00 INFO 68: Client 127.0.0.1:56863 authenticated as <ud>@ns_server</ud>
      2020-07-21T13:52:52.985601-07:00 INFO 68: HELO [proxy] XATTR, Snappy, JSON, Collections [ 127.0.0.1:56863 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]
      2020-07-21T13:52:53.023665-07:00 INFO (WAREHOUSE) EventuallyPersistentEngine::dcpOpen: opening new DCP Consumer handler - stream name:replication:ns_1@172.23.97.74->ns_1@172.23.120.81:WAREHOUSE, opaque:0, seqno:0, flags:0b00000000000000000000000100100100 value:{"consumer_name":"ns_1@172.23.120.81"}
      2020-07-21T13:52:53.023690-07:00 INFO 68: DCP connection opened successfully. INCLUDE_XATTRS, DELETE_TIMES, INCLUDE_DELETED_USER_XATTRS [ 127.0.0.1:56863 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]
      2020-07-21T13:52:53.043747-07:00 INFO 68: (WAREHOUSE) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.97.74->ns_1@172.23.120.81:WAREHOUSE - (vb:1023) Attempting to add stream: opaque_:1, start_seqno_:0, end_seqno_:18446744073709551615, vb_uuid:162031797748993, snap_start_seqno_:0, snap_end_seqno_:0, last_seqno:0, stream_req_value:{"uid":"0"}
      2020-07-21T13:52:53.056771-07:00 WARNING 68: exception occurred in runloop during packet execution. Closing connection: xattr::utils::check_len(1592262656) exceeds 80. Cookies: [{"aiostat":"success","connection":"[ 127.0.0.1:56863 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007f98280ef510","ewouldblock":false,"packet":{"bodylen":120,"cas":1595354098397937664,"datatype":["Snappy","Xattr"],"extlen":21,"key":"<ud>.8DE6199D-102_22050</ud>","keylen":19,"magic":"ClientRequest","opaque":1,"opcode":"DCP_DELETION","vbucket":1023},"refcount":1}]
      2020-07-21T13:52:53.056847-07:00 INFO 68: (No Engine) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.97.74->ns_1@172.23.120.81:WAREHOUSE - Removing connection [ 127.0.0.1:56863 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ]
      2020-07-21T13:52:53.056980-07:00 WARNING 68: (WAREHOUSE) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.97.74->ns_1@172.23.120.81:WAREHOUSE - (vb:1023) Setting stream to dead state, last_seqno is 371, unAckedBytes is 0, status is The stream closed early because the conn was disconnected
      2020-07-21T13:52:53.235599-07:00 INFO 69: HELO [{"a":"couchbase-java-client/${project.version} (git: ${git.commit.id.describe}, core: ${git.commit.id.describe}) (Linux/3.10.0-1062.9.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_252-b09)","i":"012959CB310744C8/000000004031D9B5"}] TCP nodelay, Mutation seqno, XATTR, XERROR, Select bucket [ 172.23.104.254:56134 - 172.23.120.81:11210 (not authenticated) ]
      2020-07-21T13:52:53.272656-07:00 INFO 69: Client 172.23.104.254:56134 authenticated as <ud>Administrator</ud>
      2020-07-21T13:52:53.280790-07:00 INFO 66: [ 127.0.0.1:44114 - 127.0.0.1:11209 (<ud>@ns_server</ud>) ] Updated cluster configuration for bucket [WAREHOUSE]. New revision: 2262
      2020-07-21T13:52:53.280872-07:00 INFO Pushing new cluster config for bucket:[WAREHOUSE] revision:[2262]
      2020-07-21T13:52:53.285271-07:00 INFO 69: HELO [{"a":"couchbase-java-client/${project.version} (git: ${git.commit.id.describe}, core: ${git.commit.id.describe}) (Linux/3.10.0-1062.9.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_252-b09)","i":"323CC12376D83AC6/FFFFFFFFDC9D09FF"}] TCP nodelay, Mutation seqno, XATTR, XERROR, Select bucket [ 172.23.104.254:46704 - 172.23.120.81:11210 (not authenticated) ]
      2020-07-21T13:52:53.306776-07:00 INFO 69: Client 172.23.104.254:46704 authenticated as <ud>Administrator</ud>
      

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            drigby Dave Rigby added a comment -

            Duplicate of MB-40370 which is fixed as of 7.0.0-2647.

            Show
            drigby Dave Rigby added a comment - Duplicate of MB-40370 which is fixed as of 7.0.0-2647.

              People

              Assignee:
              mihir.kamdar Mihir Kamdar
              Reporter:
              mihir.kamdar Mihir Kamdar
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty