Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58584

[System Test] :- 'ns_server' exited with status 3

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 7.2.4
    • 7.2.1
    • ns_server
    • Enterprise Edition 7.2.1 build 5933
    • Untriaged
    • Centos 64-bit
    • 0
    • Yes

    Description

      Script to Repro

      ./sequoia -client 172.23.104.254:2375 -provider file:centos_third_cluster.yml -test tests/integration/7.2/test_7.2.yml -scope tests/integration/7.2/scope_7.2_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.2.1-5933 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true
      

      Saw this issue almost 3 days post longevity run.

      172.23.108.141:

      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2023-09-08T17:10:05.412-07:00,ns_1@172.23.108.141<0.534.0>:ns_log:consume_log:76]Service 'ns_server' exited with status 3. Restarting. Messages:
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2023-09-08T17:10:05.430-07:00,ns_1@172.23.108.141<0.534.0>:ns_log:consume_log:76]Service 'ns_server' exited with status 3. Restarting. Messages:
      /opt/couchbase/var/lib/couchbase/logs/info.log:[ns_server:info,2023-09-08T17:10:05.431-07:00,ns_1@172.23.108.141:ns_log<0.527.0>:ns_log:is_duplicate_log:156]suppressing duplicate log ns_log:0([<<"Service 'ns_server' exited with status 3. Restarting. Messages:\n[os_mon] cpu supervisor port (cpu_sup): Erlang has closed\nCrap error:{badmatch,\n {error,\n {{shutdown,\n {failed_to_start_child,ns_server_cluster_sup,\n {shutdown,\n {failed_to_start_child,ns_config_sup,\n {shutdown,\n {failed_to_start_child,ns_config,\n {noproc,\n {gen_server,call,\n [{encryption_service,'babysitter_of_ns_1@cb.local'},\n {decrypt,\n <<0,1,213,185,107,111,135,223,122,55,110,96,189,112,\n 56,207,143,131,248,166,190,208,177,188,16,68,110,\n 125,97,201,61>>},\n infinity]}}}}}}}},\n {ns_server,start,[normal,[]]}}}}\n[{ns_bootstrap,start,0,[{file,\"src/ns_bootstrap.erl\"},{line,31}]},\n {child_erlang,do_child_start,1,[{file,\"src/child_erlang.erl\"},{line,111}]},\n {child_erlang,child_start,1,[{file,\"src/child_erlang.erl\"},{line,89}]},\n {init,start_em,1,[]},\n {init,do_boot,3,[]}]">>]) because it's been seen 1 times in the past 0.018163 secs (last seen 0.018163 secs ago
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2023-09-08T17:10:05.437-07:00,ns_1@172.23.108.141<0.534.0>:ns_log:consume_log:76]Service 'ns_server' exited with status 3. Restarting. Messages:
      /opt/couchbase/var/lib/couchbase/logs/info.log:[ns_server:info,2023-09-08T17:10:05.437-07:00,ns_1@172.23.108.141:ns_log<0.527.0>:ns_log:is_duplicate_log:156]suppressing duplicate log ns_log:0([<<"Service 'ns_server' exited with status 3. Restarting. Messages:\n[os_mon] cpu supervisor port (cpu_sup): Erlang has closed\nCrap error:{badmatch,\n {error,\n {{shutdown,\n {failed_to_start_child,ns_server_cluster_sup,\n {shutdown,\n {failed_to_start_child,ns_config_sup,\n {shutdown,\n {failed_to_start_child,ns_config,\n {noproc,\n {gen_server,call,\n [{encryption_service,'babysitter_of_ns_1@cb.local'},\n {decrypt,\n <<0,1,213,185,107,111,135,223,122,55,110,96,189,112,\n 56,207,143,131,248,166,190,208,177,188,16,68,110,\n 125,97,201,61>>},\n infinity]}}}}}}}},\n {ns_server,start,[normal,[]]}}}}\n[{ns_bootstrap,start,0,[{file,\"src/ns_bootstrap.erl\"},{line,31}]},\n {child_erlang,do_child_start,1,[{file,\"src/child_erlang.erl\"},{line,111}]},\n {child_erlang,child_start,1,[{file,\"src/child_erlang.erl\"},{line,89}]},\n {init,start_em,1,[]},\n {init,do_boot,3,[]}]">>]) because it's been seen 2 times in the past 0.024274 secs (last seen 0.006111 secs ago
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2023-09-08T17:10:05.445-07:00,ns_1@172.23.108.141<0.534.0>:ns_log:consume_log:76]Service 'ns_server' exited with status 3. Restarting. Messages:
      /opt/couchbase/var/lib/couchbase/logs/info.log:[ns_server:info,2023-09-08T17:10:05.446-07:00,ns_1@172.23.108.141:ns_log<0.527.0>:ns_log:is_duplicate_log:156]suppressing duplicate log ns_log:0([<<"Service 'ns_server' exited with status 3. Restarting. Messages:\n[os_mon] cpu supervisor port (cpu_sup): Erlang has closed\nCrap error:{badmatch,\n {error,\n {{shutdown,\n {failed_to_start_child,ns_server_cluster_sup,\n {shutdown,\n {failed_to_start_child,ns_config_sup,\n {shutdown,\n {failed_to_start_child,ns_config,\n {noproc,\n {gen_server,call,\n [{encryption_service,'babysitter_of_ns_1@cb.local'},\n {decrypt,\n <<0,1,213,185,107,111,135,223,122,55,110,96,189,112,\n 56,207,143,131,248,166,190,208,177,188,16,68,110,\n 125,97,201,61>>},\n infinity]}}}}}}}},\n {ns_server,start,[normal,[]]}}}}\n[{ns_bootstrap,start,0,[{file,\"src/ns_bootstrap.erl\"},{line,31}]},\n {child_erlang,do_child_start,1,[{file,\"src/child_erlang.erl\"},{line,111}]},\n {child_erlang,child_start,1,[{file,\"src/child_erlang.erl\"},{line,89}]},\n {init,start_em,1,[]},\n {init,do_boot,3,[]}]">>]) because it's been seen 3 times in the past 0.033077 secs (last seen 0.008803 secs ago...
      

      This was not seen on the run we had on 7.2.1-5932. Hence marking it a regression. I don't believe this is a regression though, probably some one off issue.

      Saw some CBSE's and bugs (see MB-56447) which had similar messages. Not really sure if its related to it though. cbcollect_info attached.

      Update
      Actually looks more like a dup of MB-56471.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Abhijeeth.Nuthan Abhijeeth Nuthan
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty