Minidump is 2d18f71d-1efb-4906-9072-462523dcb9b5.dmp on 172.23.106.120.
Not sure how to get the bt full on windows, so for now just attaching cbcollects. Logs seems to have rolled over.
{[{<<"[*** LOG ERROR ***] [2021-04-20 11:39:38] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:38:36] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:37:36] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:36:17] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:35:16] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:34:16] [globalBucketLogger] argument index out of range">>,
90}],
[{<<"W0420 11:04:29.689070 3228 HazptrDomain.h:671] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object">>,
236},
{<<"[*** LOG ERROR ***] [2021-04-20 11:30:16] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:31:17] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:32:17] [globalBucketLogger] argument index out of range">>,
90}]}},
undefined,undefined,[],0}
** Reason for termination ==
** {abnormal,-529697949}
Daniel Owen
added a comment - - edited On .120 see babysitter log
[error_logger:error,2021-04-20T06:41:14.272-07:00,babysitter_of_ns_1@cb.local:<0.140.0>:ale_error_logger_handler:do_log:101]
=========================ERROR REPORT=========================
** Generic server <0.140.0> terminating
** Last message in was {#Port<0.16>,{exit_status,-529697949}}
** When Server state == {state,#Port<0.16>,
{memcached,
"c:/Program Files/Couchbase/Server/bin/memcached",
["-C",
"c:/Program Files/Couchbase/Server/var/lib/couchbase/config/memcached.json"],
[{env,
[{"EVENT_NOSELECT","1"},
{"MEMCACHED_TOP_KEYS","5"},
{"CBSASL_PWFILE",
"c:/Program Files/Couchbase/Server/var/lib/couchbase/isasl.pw"},
{"JE_MALLOC_CONF","narenas:1"}]},
use_stdio,stderr_to_stdout,exit_status,
stream]},
{ringbuffer,1046,1024,
{[{<<"[*** LOG ERROR ***] [2021-04-20 11:39:38] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:38:36] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:37:36] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:36:17] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:35:16] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:34:16] [globalBucketLogger] argument index out of range">>,
90}],
[{<<"W0420 11:04:29.689070 3228 HazptrDomain.h:671] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object">>,
236},
{<<"[*** LOG ERROR ***] [2021-04-20 11:30:16] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:31:17] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:32:17] [globalBucketLogger] argument index out of range">>,
90}]}},
undefined,undefined,[],0}
** Reason for termination ==
** {abnormal,-529697949}
This bug is hit after multi node swap rebalance(2 nodes) along with durability data load. Then we notice that 172.23.106.120 never comes back up again(http://node:8091 was not accessible). To do a cbollect for this bug I had to reboot the box to make it work again.
Balakumaran Gopal
added a comment -
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444555555555.ini rerun=False,get-cbcollect-info=True,crash_warning=True,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE'
This bug is hit after multi node swap rebalance(2 nodes) along with durability data load. Then we notice that 172.23.106.120 never comes back up again( http://node:8091 was not accessible). To do a cbollect for this bug I had to reboot the box to make it work again.
Therefore resovling as an environmental issue. Box only hag 6GB of RAM
[warn] Installed RAM : 6140 MB - less than recommended (16384 MB)
Therefore will resolve as "Not a Bug" and suggest running with more memory.
Changed my mind that this is may be a bug (although might not be with KV)
Daniel Owen
added a comment - - edited Therefore resovling as an environmental issue. Box only hag 6GB of RAM
[warn] Installed RAM : 6140 MB - less than recommended (16384 MB)
Therefore will resolve as "Not a Bug" and suggest running with more memory.
Changed my mind that this is may be a bug (although might not be with KV)
Believe this to be caused by issues highlighted in MB-45061.
Closing as incomplete.
If issues persist after MB-45061 is resolved then please reopen.
Daniel Owen
added a comment - Believe this to be caused by issues highlighted in MB-45061 .
Closing as incomplete.
If issues persist after MB-45061 is resolved then please reopen.
On .120 see babysitter log
[error_logger:error,2021-04-20T06:41:14.272-07:00,babysitter_of_ns_1@cb.local:<0.140.0>:ale_error_logger_handler:do_log:101]
=========================ERROR REPORT=========================
** Generic server <0.140.0> terminating
** Last message in was {#Port<0.16>,{exit_status,-529697949}}
** When Server state == {state,#Port<0.16>,
{memcached,
"c:/Program Files/Couchbase/Server/bin/memcached",
["-C",
"c:/Program Files/Couchbase/Server/var/lib/couchbase/config/memcached.json"],
[{env,
[{"EVENT_NOSELECT","1"},
{"MEMCACHED_TOP_KEYS","5"},
{"CBSASL_PWFILE",
"c:/Program Files/Couchbase/Server/var/lib/couchbase/isasl.pw"},
{"JE_MALLOC_CONF","narenas:1"}]},
use_stdio,stderr_to_stdout,exit_status,
stream]},
{ringbuffer,1046,1024,
{[{<<"[*** LOG ERROR ***] [2021-04-20 11:39:38] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:38:36] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:37:36] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:36:17] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:35:16] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:34:16] [globalBucketLogger] argument index out of range">>,
90}],
[{<<"W0420 11:04:29.689070 3228 HazptrDomain.h:671] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object">>,
236},
{<<"[*** LOG ERROR ***] [2021-04-20 11:30:16] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:31:17] [globalBucketLogger] argument index out of range">>,
90},
{<<"[*** LOG ERROR ***] [2021-04-20 11:32:17] [globalBucketLogger] argument index out of range">>,
90}]}},
undefined,undefined,[],0}
** Reason for termination ==
** {abnormal,-529697949}