[System Test] Port exited with status 2 - panic: expected newly cut file to have sequence:offset 2:8, got 1:8

                               {metakv,

                                   <<"/indexing/ddl/commandToken/create/13112280625137711765/1">>}]..)

[ns_server:debug,2023-07-11T07:33:40.235Z,ns_1@svc-qi-node-005.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com:ns_config_log<0.264.0>:ns_config_log:log_common:275]config change:

{local_changes_count,<<"f2770329bbec0afedc07c76fb91fe7ab">>} ->

[{'_vclock',[{<<"f2770329bbec0afedc07c76fb91fe7ab">>,{942,63856280020}}]}]

[ns_server:debug,2023-07-11T07:33:41.674Z,ns_1@svc-qi-node-005.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com:prometheus-goport<0.2226.0>:goport:handle_eof:585]Stream 'stderr' closed

[ns_server:debug,2023-07-11T07:33:41.674Z,ns_1@svc-qi-node-005.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com:prometheus-goport<0.2226.0>:goport:handle_eof:585]Stream 'stdout' closed

[ns_server:info,2023-07-11T07:33:41.675Z,ns_1@svc-qi-node-005.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com:prometheus-goport<0.2226.0>:goport:handle_process_exit:566]Port exited with status 2.

[error_logger:error,2023-07-11T07:33:41.765Z,ns_1@svc-qi-node-005.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com:<0.2220.0>:ale_error_logger_handler:do_log:101]

=========================ERROR REPORT=========================

** Generic server <0.2220.0> terminating 

** Last message in was {<0.2226.0>,{exit_status,2}}

** When Server state == {state,<0.2226.0>,3531,

                            {prometheus,"/opt/couchbase/bin/prometheus",

                                ["--config.file",

                                 "/opt/couchbase/var/lib/couchbase/config/prometheus.yml",

                                 "--web.enable-admin-api",

                                 "--web.enable-lifecycle",

                                 "--storage.tsdb.retention.size","1024MB",

                                 "--storage.tsdb.retention.time","365d",

                                 "--web.listen-address","127.0.0.1:9123",

                                 "--storage.tsdb.max-block-duration","25h",

                                 "--storage.tsdb.path",

                                 "/opt/couchbase/var/lib/couchbase/stats_data",

                                 "--log.level","debug","--query.max-samples",

                                 "200000","--storage.tsdb.no-lockfile",

                                 "--query.lookback-delta","600s",

                                 "--web.config.file",

                                 "/opt/couchbase/var/lib/couchbase/config/prometheus_auth"],

                                [via_goport,exit_status,stderr_to_stdout,

                                 {env,[]}]},

                            {ringbuffer,1086,1024,

                                {[{<<")\n\t/home/couchbase/jenkins/workspace/cbdeps-platform-build/goproj/src/github.com/prometheus/prometheus/tsdb/chunks/chunk_write_queue.go:94 +0x14a\ncr"...>>,

                                   370},

                                  {<<"github.com/prometheus/prometheus/tsdb/chunks.(*chunkWriteQueue).processJob(0xc00081c180, {0x1, 0xd, 0x18943c6aebb, 0x18943dea389, {0x35e7538, 0x"...>>,

                                   402}],

                                 [{<<"panic: expected newly cut file to have sequence:offset 2:8, got 1:8\n\ngoroutine 432 [running]:\ngithub.com/prometheus/prometheus/tsdb.handleChunkW"...>>,

                                   314}]}},

                            prometheus,undefined,[],0}

** Reason for termination ==

** {abnormal,2}

[error_logger:error,2023-07-11T07:33:41.766Z,ns_1@svc-qi-node-005.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com:<0.2220.0>:ale_error_logger_handler:do_log:101]

=========================CRASH REPORT=========================

  crasher:

    initial call: ns_port_server:init/1

    pid: <0.2220.0>

    registered_name: []

    exception exit: {abnormal,2}

      in function  gen_server:handle_common_reply/8 (gen_server.erl, line 811)

    ancestors: [prometheus_cfg,ns_server_sup,ns_server_nodes_sup,<0.2118.0>,

                  ns_server_cluster_sup,root_sup,<0.145.0>]

    message_queue_len: 1

    messages: [{'EXIT',<0.2226.0>,normal}]

    links: [<0.2214.0>]

    dictionary: []

    trap_exit: true

    status: running

    heap_size: 17731

    stack_size: 29

    reductions: 40191

  neighbours:

https://cb-engineering.s3.amazonaws.com/SysTest_11Jul_Slow_Indexing/collectinfo-2023-07-12T040332-ns_1%40svc-d-node-001.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com.zip

https://cb-engineering.s3.amazonaws.com/SysTest_11Jul_Slow_Indexing/collectinfo-2023-07-12T040332-ns_1%40svc-d-node-002.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com.zip

https://cb-engineering.s3.amazonaws.com/SysTest_11Jul_Slow_Indexing/collectinfo-2023-07-12T040332-ns_1%40svc-d-node-003.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com.zip

https://cb-engineering.s3.amazonaws.com/SysTest_11Jul_Slow_Indexing/collectinfo-2023-07-12T040332-ns_1%40svc-qi-node-004.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com.zip

https://cb-engineering.s3.amazonaws.com/SysTest_11Jul_Slow_Indexing/collectinfo-2023-07-12T040332-ns_1%40svc-qi-node-005.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com.zip

On 004

[ns_server:info,2023-07-11T07:33:41.908Z,ns_1@svc-qi-node-004.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com:prometheus-goport<0.2168.0>:goport:handle_process_exit:566]Port exited with status 2.

On 002

[ns_server:info,2023-07-11T07:33:40.952Z,ns_1@svc-d-node-002.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com:prometheus-goport<0.2444.0>:goport:handle_process_exit:566]Port exited with status 2.